Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Many organizations and industries continuously struggle to achieve a foundation of a sound verification and validation process. Because verification and validation is only marginally addressed in software process improvement models like CMMI, a separate verification and validation improvement model has been developed. The framework of the improvement model presented in this paper is based on the Testing Maturity Model (TMM). However considerable enhancements are provided, such as a V&V assessment procedure and a detailed metrics program to determine effectiveness and efficiency of the improvements. This paper addresses the development approach of MB-V2M2, an outline of the model, early validation experiments, current status and future outlook. The resulting model is dubbed MB-V2M2 (Metrics Based Verification@Validation Maturity Model). <s> BIB001 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> It is well known that most of mission critical systems like defense systems demand higher quality than general software systems. In order to assure the high quality of the systems, those software systems must be thoroughly tested with well defined testing processes. As software systems evolve, software testing should also be improved with continual-base. Even if there are some previous researches on test process model, those models have some limitations and problems to be applied for software testing process improvement in a specific domain like military weapon systems. In this paper, we propose a new testing maturity model, called MND-TMM (Ministry Of National Defense-Testing Maturity Model), which can be adopted effectively for the weapon software system development. We also present OWL ontology for the MND-TMM, called MTO(MND-TMM ontology) to help software organizations set up their effective testing strategies and improve their testing processes, which can lead to high quality product developments. <s> BIB002 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Existing test process assessment and improvement models intend to raise maturity of an organization with reference to testing activities. Such process assessments are based on ldquowhatrdquo testing activities are being carried out, and thus implicitly evaluate process quality. Other test process measurement techniques attempt to directly assess some partial quality attribute such as efficiency or effectiveness some test metrics. There is a need for a formalized method of test process quality evaluation that addresses both implicitly and partially of these current evaluations. This paper describes a conceptual framework to specify and explicitly evaluate test process quality aspects. The framework enables provision of evaluation results in the form of objective assessments, and problem-area identification to improve the software testing processes. <s> BIB003 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Software testing is a core activity in quality assurance. To improve the test process, we can use best practice models which describe in detail what to do in organizational test processes. The improvement activities using best practice models are performed as follows: checking the current status of test processes, suggesting and planning new actions, and implementing the actions. However, it is difficult to apply all of these actions to the organization due to the limitation of resources. In this paper, we suggest a strategy for optimizing test process action plans. The background of this research is TMM (Testing Maturity Model), which is the most representative test process model. By applying design of experiments to the TMM assessment procedure, we can accept the actions selectively by statistical significance and find the best solution. <s> BIB004 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Embedded software requires high quality to meet customer requirements and to achieve product competitiveness. Testability Maturity Model (TMM)and Test Process Improvement (TPI) are recommended to improve test activities, which are important for product quality. However, TMM and TPI in themselves are not appropriate for embedded software since they are lack of consideration on embedded software. This paper deals with the properties of embedded software and industrial challenges based on existing model and we suggest Embedded Test Process Improvement Model (Emb-TPI). Emb-TPI considers practical improvement strategies for the quality of embedded software and focuses on evaluation procedures that achieve cost-effectiveness and valid evaluation results.We present the results of applying Emb-TPI to product development division (PDD) to prove that the model is valid. Emb-TPI provides methods to evaluate test activities in embedded software development and helps to reveal insufficient areas of test activities, which ultimately improves test capabilities. <s> BIB005 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> As one of the most challenging and costly processes, the software testing provides strong support for the high quality software. The popular process improvement models do not adequately address testing application issues, nor has the knowledge management (KM) implementation in testing been well defined. The plan-do-check-action (PDCA)-based testing improvement framework is proposed to address these issues. The framework contains the adaptive processes definition, the processes implementation and measurement analysis, and benchmarking-based assessment approach. Experiments show that: the framework can stimulate team member's subjective initiative, promote the software testing process and improve the testing service quality. <s> BIB006 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> One of the latest additions in defining the test process is the upcoming ISO/IEC 29119 standard, which aims to define a universally applicable generic test process model. However, currently the standard does not offer any support for the adoption process of the model. In this paper, we present our framework, which aims to combine a maturity level-based approach with the standard process. Our objective was to create an easy-to-use framework for organizations to assess how closely their existing test process follows the standard, and give feedback on improvement objectives. Our results indicate that combining maturity levels with the standard is a viable approach to assess the implementation of the standard in practice. <s> BIB007 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Automated test generation is gaining popularity in the software industry, largely due to its labor-saving benefits and its ability to achieve high test coverage. The introduction of this technology into an organization does not, however, always meet with success. One reason for this is often the fact that the testing process and organization are not adjusted accordingly. Thus, in order for an organization to successfully pursue automated test generation, the test process must also be improved to enable this development. In this paper, we introduce an automated test generation add-on for the popular test process improvement model, TPI. We also present a baseline TPI profile for successful introduction of automated test generation. <s> BIB008 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Testing is an essential part of the software development lifecycle for gaining trust in the quality of the delivered product. Concerns have been raised over the maturity of existing test processes and techniques, and the desire to improve has been acknowledged. Even though there are test process improvement models available on the market, the guidelines on how to use them are unsatisfactory. This paper describes the empirical analysis of Test Maturity Model integration (TMMi) through a single-object case study of test process assessment and improvement planning conducted in Playtech Estonia’s Casino unit. An evaluation on the performance of TMMi is presented, raising also some concerns over its applicability in agile environments. Improvement possibilities for TMMi are described, which contribute to the potential enhancement of the framework. <s> BIB009 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Context: Testing is one of the most important phases of software development. However, in industry this phase is usually compromised by the lack of planning and resources. Due to it, the adoption of a streamlined testing process can lead to the construction of software products with desirable quality levels. Objective: Presenting the results of a survey conducted to identify a set of key practices to support the definition of a generic, streamlined software testing process, based on the practices described in the TMMi (Test Maturity Model integration). Method: Based on the TMMi, we have performed a survey among software testing professionals who work in both academia and industry. Their responses were analyzed quantitatively and qualitatively in order to identify priority practices to build the intended generic process. Results: The analysis enabled us to identify practices that were ranked as mandatory, those that are essential and should be implemented in all cases. This set of practices (33 in total) represents only 40% of the TMMis full set of practices, which sums up to 81 items related to the steps of a testing process. Conclusion: The results show that there is a consensus on a subset of practices that can guide the definition of a lean testing process when compared to a process that includes all TMMi practices. It is expected that such a process encourages a wider adoption of testing activities within the software industry. <s> BIB010 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> In this paper, we report about a systematic mapping study in software requirements prioritization with a specific focus on empirical studies.The results show that the interest from the research community is clustered around the more recent years. The majority of the studies are about the validation of research or solution proposals. We report the prevalence of studies on techniques and methodologies while there is a scarce interest in the strict evaluation of tools that could be beneficial to industry. In most of the empirical studies we found a bottom-up approach, centering on the techniques and on accuracy as the dependent variable, as well as on functional requirements as the main research focus. Based on the results, we provide recommendations for future research directions. <s> BIB011 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> In agile software development, industries are becoming more dependent on automated test suites. Thus, the test code quality is an important factor for the overall system quality and maintainability. We propose a Test Automation Improvement Model (TAIM) defining ten key areas and one general area. Each area should be based on measurements, to fill the gap of existing assessments models. The main contribution of this paper is to provide the outline of TAIM and present our intermediate results and some initial metrics to support our model. Our initial target has been the key area targeting implementation and structure of test code. We have used common static measurements to compare the test code and the source code of a unit test automation suite being part of a large complex telecom subsystem. Our intermediate results show that it is possible to outline such an improvement model and our metrics approach seems promising. However, to get a generic useful model to aid test automation evolution and provide for comparable measurements, many problems still remain to be solved. TAIM can as such be viewed as a framework to guide the research on metrics for test automation artifacts. <s> BIB012 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> Model-based testing (MBT) provides a systematic and automated way to facilitate rigorous testing of software systems. MBT has been an intense area of research and a large number of MBT techniques have been developed in the literature and in the practice. However, all of the techniques have been developed using their own concepts and terminology of MBT, which are very often different than other techniques and at times have conflicting semantics. Moreover, while working on MBT projects with our industrial partners in the last several years, we were unable to find a unified way of defining MBT techniques based on standard terminology. To precisely define MBT concepts with the aim of providing common understanding of MBT terminology across techniques, we formalize a small subset of the recently released ISO/IEC/IEEE 29119 Software Testing Standard as a conceptual model (UML class diagrams) together with OCL constraints. The conceptual model captures all the necessary concepts based on the standard terminology that are mandatory or optional in the context of MBT techniques and can be used to define new MBT tools and techniques. To validate the conceptual model, we instantiated its concepts for various MBT techniques previously developed in the context of our industrial partners. Such instantiation automatically enforces the specified OCL constraints. This type of validation provided us feedback to further refine the conceptual model. Finally, we also provide our experiences and lessons learnt for such formalization and validation. <s> BIB013 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> , TMM [TM1], TPI [TM3], TIM [TM9]) <s> The software testing phase, despite its importance, is usually compromised by the lack of planning and resources in industry. This can risk the quality of the derived products. The identification of mandatory testing-related practices may lead to the definition of feasible processes for varied sizes of software companies. In this context, this work aims at identifying a set of key practices to support the definition of a generic, streamlined software testing process, based on practices that are described in the TMMi (Test Maturity Model integration), and verify the alignment of the devised process with the TMMi levels. We have performed a survey amongst Brazilian software testing professionals who work in both academia and industry, in order to identify priority practices to build the intended, streamlined process; additionally, we applied a diagnosis tool in order to measure the level of TMMi which is fulfilled with the devised process. A set of 33 (out of 81) practices were ranked as mandatory by most of participants, which represents 40 % of the TMMi’s full set of practices; on the downside, a testing process that relies on this subset of TMMi practices does not fully fulfil level 2 (Managed) of the maturity model. The identified subset of practices can guide the definition of a lean testing process when compared to a process that includes all TMMi practices; it is expected that such a process encourages a wider adoption of testing activities in software development; even though the streamlined process does not encompass many practices that are spread across TMMi levels, a substantial subset of level 2 practices (Managed) should be accomplished with its adoption. <s> BIB014
and their usability for SME. Although the models are applicable in general, organizations usually consider that models TM3: Developed based on the practical knowledge and experiences BIB002 . Complementarity TM1: A counterpart of the CMM model BIB001 . TM2: A complement model to CMMI; helps to perform any evaluations according to CMMI BIB009 . Model Structure TM2: The evolutionary staged model allows assessment of the current state of maturity and suggests a clear improvement path BIB009 . TM3: It allows to achieve different Key Areas simultaneously BIB008 . TM9: Key Areas are assessed separately and are relatively independent BIB007 . TM11: Independent from the development life cycle and any testing process being followed BIB003 . TM17: The small steps are intended to provide an easier path to follow  . Applicability TM3: Appropriate for specific test projects because it is naturally technical BIB008 . TM8: Can be applied to different types of testing techniques BIB013 . Other TM12: It focuses also on individual's subjective initiative and competitive advantage BIB006 . TM17: Compared to other models, MTPF provides the low threshold for introducing each phase  . are not suitable for SME and furthermore insufficient for specific domains such as defense or embedded software implementation, because they are too extensive. Moreover, the resources needed for the adaptation of the models are not available in smaller organizations. In that case, SMEs would need to make a lot of effort to establish model's processes  . The empirical study reviewing TMMi through a survey published by Camargo et al. BIB014 , extends the results presented in BIB010 . The authors observed that TMMi and MPT.BR have too many practices which have to be satisfied and not all of them are applicable for smaller-sized companies. Furthermore, they do not define priorities in case of absence of time or resources. Lack of support for specific areas. Prioritization plays a major role in software engineering BIB011 . Several specific areas such as prioritizing or test automation are not supported by some models. The case study of Jung BIB005 performed analysis of existing testing models with the aim to find a suitable model for embedded software. The authors discuss disadvantages of the TMM [TM1] and TPI [TM3] models and provide reasons why they are insufficient for embedded software which requires high quality. Based on their findings, test elementary factors are not taken into account in TMM, specifically test organization, human resource management and test bed. Moreover, TMM does not describe procedures and guidelines for process evaluation and improvement in detail BIB005 . The TPI model also does not meet the embedded software requirements, because it does not provide practical and realistic improvements on that kind of software BIB005 . The weaknesses of TMM [TM1] are mentioned also in the study of Oh et al. BIB004 , which highlights the issue of prioritizing multiple actions. The paper deals with an effective implementation strategy for the models and it includes also ranking importance of improvements activities. However, TMM does not provide any procedure for prioritizing multiple actions which focus on the same maturity goals. This drawback can lead to failure of the activities and wasting the organizational resources without any positive effects BIB004 . The case study of Ryu et al. BIB002 includes an initial part about reviewing the testing models (e.g., TMM [TM1], TPI [TM3], TIM [TM9]), their characteristics and limitations. First, the study describes TMM and its usage as a supplementary model of the CMM. According to this relation, it is more effective to apply them together, while based on the study it is difficult to use only the TMM or even combine with other models BIB002 . Other drawbacks of TMM are lack of important key areas such as Test Environment, Office Environment, Reporting, Defect Management, and Testware Management and high complexity of included key areas, that makes them difficult to achieve BIB002 . The study of Camargo et al. BIB010 presents a survey performed among software testing professionals. The paper reports several limitations of TMMi [TM2] based on the results from the review. According to the review the model does not define any priorities of activities or dependencies between the process areas BIB010 . Eldh et al. BIB012 comment in their study that although TMMi is adapted for new processes (e.g. agile), it does not provide improvement steps for real test automation advancement. The study of Rungi and Matulevičius BIB009 , which mentions TMMi weaknesses, indicates that the model does not include detailed documentation about agile practices in comparison with CMMI and the application of best practices. Model complexity. About the structure and features of the TPI model, the authors consider incompatibility of TPI with the CMM / CMMI and TMM caused by different structure of levels, and missing checking of items for stateof-the-art testing techniques after their publishing BIB002 . In general, these models are too complex and difficult for organizations to apply and because of many procedures and process areas they require the process assessment from experts to inspect the gaps BIB002 . The study of Camargo et al. BIB010 considers the TMMi model structure as very comprehensive, therefore it can be confusing and demanding to understand differences between some activities. The study of Ali and Yue BIB013 uses ISO/IEC/IEEE 29119 [TM8] as a conceptual model for Model-Based Testing techniques. Based on their experience, the model has very high abstraction level, that can lead to misunderstandings about the concepts. The authors solve this issue by developing the conceptual model and subsequently demonstrating it in the context of different types of testing techniques. Practical limitations. Based on the experience with MTPF implementation within an organization and a performed survey in other companies, the authors conclude the model as applicable in SMEs but requiring continuous adaptation of the practices in order for them to stay current, effective and useful  . At the beginning, the study of Kasurinen et al. BIB007 discusses practical limitations of TMMi application in real-life companies. Specifically, the idea of moving all processes to one level before achieving the next one is not feasible, and the level requirements are counter-intuitive. Others. The PDCA-based software testing improvement framework [TM12] is developed and published by XuXiang and Zhang BIB006 . The framework is designed to address the issues of testing application and knowledge management (KM) implementation by focusing mostly on agile process, benchmarking process and KM. Although the framework is flexible and efficient, the authors defined several constraints which can influence the effect. First, organizations should construct and support the knowledgeoriented culture and sustain the enthusiasm from the teams. Second, during planning and adaptation of the testing activities, organizations should understand popular testing models completely. And third, KM has to be included at the organizational level and managers within the organizations should support it BIB006 . Although the Self-Assessment framework for ISO / IEC / IEEE 29119 based on TIM [TM15] suggested by Kasurinen et al. BIB007 can become a feasible tool for defining process improvement objectives, it needs further development to be validated as a testing model for real cases.