Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> Many organizations and industries continuously struggle to achieve a foundation of a sound verification and validation process. Because verification and validation is only marginally addressed in software process improvement models like CMMI, a separate verification and validation improvement model has been developed. The framework of the improvement model presented in this paper is based on the Testing Maturity Model (TMM). However considerable enhancements are provided, such as a V&V assessment procedure and a detailed metrics program to determine effectiveness and efficiency of the improvements. This paper addresses the development approach of MB-V2M2, an outline of the model, early validation experiments, current status and future outlook. The resulting model is dubbed MB-V2M2 (Metrics Based Verification@Validation Maturity Model). <s> BIB001 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> It is well known that most of mission critical systems like defense systems demand higher quality than general software systems. In order to assure the high quality of the systems, those software systems must be thoroughly tested with well defined testing processes. As software systems evolve, software testing should also be improved with continual-base. Even if there are some previous researches on test process model, those models have some limitations and problems to be applied for software testing process improvement in a specific domain like military weapon systems. In this paper, we propose a new testing maturity model, called MND-TMM (Ministry Of National Defense-Testing Maturity Model), which can be adopted effectively for the weapon software system development. We also present OWL ontology for the MND-TMM, called MTO(MND-TMM ontology) to help software organizations set up their effective testing strategies and improve their testing processes, which can lead to high quality product developments. <s> BIB002 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> Existing test process assessment and improvement models intend to raise maturity of an organization with reference to testing activities. Such process assessments are based on ldquowhatrdquo testing activities are being carried out, and thus implicitly evaluate process quality. Other test process measurement techniques attempt to directly assess some partial quality attribute such as efficiency or effectiveness some test metrics. There is a need for a formalized method of test process quality evaluation that addresses both implicitly and partially of these current evaluations. This paper describes a conceptual framework to specify and explicitly evaluate test process quality aspects. The framework enables provision of evaluation results in the form of objective assessments, and problem-area identification to improve the software testing processes. <s> BIB003 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> As one of the most challenging and costly processes, the software testing provides strong support for the high quality software. The popular process improvement models do not adequately address testing application issues, nor has the knowledge management (KM) implementation in testing been well defined. The plan-do-check-action (PDCA)-based testing improvement framework is proposed to address these issues. The framework contains the adaptive processes definition, the processes implementation and measurement analysis, and benchmarking-based assessment approach. Experiments show that: the framework can stimulate team member's subjective initiative, promote the software testing process and improve the testing service quality. <s> BIB004 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> One of the latest additions in defining the test process is the upcoming ISO/IEC 29119 standard, which aims to define a universally applicable generic test process model. However, currently the standard does not offer any support for the adoption process of the model. In this paper, we present our framework, which aims to combine a maturity level-based approach with the standard process. Our objective was to create an easy-to-use framework for organizations to assess how closely their existing test process follows the standard, and give feedback on improvement objectives. Our results indicate that combining maturity levels with the standard is a viable approach to assess the implementation of the standard in practice. <s> BIB005 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> Automated test generation is gaining popularity in the software industry, largely due to its labor-saving benefits and its ability to achieve high test coverage. The introduction of this technology into an organization does not, however, always meet with success. One reason for this is often the fact that the testing process and organization are not adjusted accordingly. Thus, in order for an organization to successfully pursue automated test generation, the test process must also be improved to enable this development. In this paper, we introduce an automated test generation add-on for the popular test process improvement model, TPI. We also present a baseline TPI profile for successful introduction of automated test generation. <s> BIB006 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> Testing is an essential part of the software development lifecycle for gaining trust in the quality of the delivered product. Concerns have been raised over the maturity of existing test processes and techniques, and the desire to improve has been acknowledged. Even though there are test process improvement models available on the market, the guidelines on how to use them are unsatisfactory. This paper describes the empirical analysis of Test Maturity Model integration (TMMi) through a single-object case study of test process assessment and improvement planning conducted in Playtech Estonia’s Casino unit. An evaluation on the performance of TMMi is presented, raising also some concerns over its applicability in agile environments. Improvement possibilities for TMMi are described, which contribute to the potential enhancement of the framework. <s> BIB007 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Models' advantages <s> Model-based testing (MBT) provides a systematic and automated way to facilitate rigorous testing of software systems. MBT has been an intense area of research and a large number of MBT techniques have been developed in the literature and in the practice. However, all of the techniques have been developed using their own concepts and terminology of MBT, which are very often different than other techniques and at times have conflicting semantics. Moreover, while working on MBT projects with our industrial partners in the last several years, we were unable to find a unified way of defining MBT techniques based on standard terminology. To precisely define MBT concepts with the aim of providing common understanding of MBT terminology across techniques, we formalize a small subset of the recently released ISO/IEC/IEEE 29119 Software Testing Standard as a conceptual model (UML class diagrams) together with OCL constraints. The conceptual model captures all the necessary concepts based on the standard terminology that are mandatory or optional in the context of MBT techniques and can be used to define new MBT tools and techniques. To validate the conceptual model, we instantiated its concepts for various MBT techniques previously developed in the context of our industrial partners. Such instantiation automatically enforces the specified OCL constraints. This type of validation provided us feedback to further refine the conceptual model. Finally, we also provide our experiences and lessons learnt for such formalization and validation. <s> BIB008
There are several advantages that are reported for each of the reviewed models in the empirical studies (Table 9) . We grouped them into five categories: practice, complementarity, model structure, applicability and other. Practice. As an advantage of two models is considered their practical experience. The case study of Jacobs and Trienekens BIB001 discusses strong points of TMM [TM1] as a baseline of the MB-VV-MM development (TM13): specifically that TMM reflects over forty years of software testing growth in the industry BIB001 . Based on the study of Ryu et al. BIB002 , the TPI model [TM3] provides guidelines for maturity level assessment, which are inspired by practical experience of the test process in organizations. Complementarity. According to the study of Jacobs and Trienekens BIB001 , the TMM model can be used to complement CMM, which does not sufficiently deal with testing issues BIB001 . Rungi and Matulevičius [61] analyze the TMMi model [TM2] and its benefits. They consider TMMi as a complementary model to CMMI: experience and results of the TMMi assessment procedure can be used to perform any future evaluation according to CMMI BIB007 . Model Structure. Several advantages come from the models' structure. The research of Rungi and Matulevičius BIB007 also discusses the structure of the TMMi model [TM2] . The evolutionary staged model of TMMi maturity levels allows evaluation of the current status and provides a clear improvement path for achievement of a higher maturity level BIB007 . Furthermore, Heiskanen et al. BIB006 recognize benefits of TPI for the ATG add-on for TPI [TM16] development. One of the strengths of TPI that the authors report is the possibility to progress simultaneously in many different Key Areas, which allows organizations to focus on more selected areas BIB006 . The significant characteristic of the TIM [TM9] model discussed in the case study of Kasurinen et al. BIB005 is a separate assessment and independence of the key areas. Organizations can better understand what areas need to be improved BIB005 . The study of Farooq et al. BIB003 focuses on a theoretical and conceptual representation of the Meta-measurement approach [TM11] . It evaluates the model as a lightweight framework, with the strength of the independence from the development life cycle as well as from any testing process in use. One of the strengths of MTPF [TM17] discussed in the study of Karlström et al.  is the specification of small steps, which allow to follow an easier improvement path. Applicability. Model applicability is one of the key points in the selecting process. According to the study of Heiskanen et al. BIB006 , TPI is suitable for specific test projects or approaches thanks to its natural technical base BIB006 . The empirical study of Ali and Yue BIB008 focuses on a conceptual model based on ISO/IEC/IEEE 29119 [TM8]. The authors consider the standard defined at a very high level of abstraction, that can help to apply it to different types of testing techniques. For that reason, they use a small subset of the standard as a conceptual model to understand terminology and define key concepts of the Model-based Testing technique. Other. Two advantages cannot be categorized because of their specialization. Xu-Xiang and Zhang BIB004 have developed the PDCA-based software testing improvement framework [TM12] to address testing application issues. Based on the analysis, the framework is shown to be flexible, easy-to-use and helps to influence the initiative of the staff. Moreover, knowledge management testing activities support the reuse of testing cases and help to raise the testing center's competitive advantage BIB004 . In comparison to other models, MTPF introduces each phase using the low threshold and ensures the validity by using rigorous and well-established research methodologies  .