Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Testing is a critical component of a mature software development process. It is one of the most challenging and costly process activities, and in its fullest definition if provides strong support for the development of high qualify software. Existing maturity models do not adequately address testing issues, nor has the concept of a mature testing process been well defined. We are developing a Testing Maturity Model (TMM) to address these issues. The TMM will contain a set of maturity levels through which an organization can progress towards testing process maturity, a set of recommended practices at each level of maturity that can be put into place, and an assessment model that will allow software development organizations to evaluate and improve their testing processes. In this paper we discuss our approach to TMM development, the major features of the TMM, its ties to the Capability Maturity Model, and our initial plans for a Testers' Tool Workbench that will support testing process maturity growth. <s> BIB001 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Judging by the title, you might expect to discover between the covers a lot of tips and techniques to really help you and your team get better at testing. Unfortunately, if you have more than a little experience as a tester, you will find little here that is new to you. Conversely, if you are relatively new to testing, many techniques in Dustin’s list of fifty might be new to you, but for the most part, you won’t get enough detail to really learn them. For me, reading this book was about as helpful as reading the menu at my favorite take-out restaurant: I’ve seen all the choices before, and Dustin did not have much new to say about them. <s> BIB002 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Objective: Our objective is to describe how softwareengineering might benefit from an evidence-basedapproach and to identify the potential difficultiesassociated with the approach.Method: We compared the organisation and technicalinfrastructure supporting evidence-based medicine (EBM)with the situation in software engineering. We consideredthe impact that factors peculiar to software engineering(i.e. the skill factor and the lifecycle factor) would haveon our ability to practice evidence-based softwareengineering (EBSE).Results: EBSE promises a number of benefits byencouraging integration of research results with a view tosupporting the needs of many different stakeholdergroups. However, we do not currently have theinfrastructure needed for widespread adoption of EBSE.The skill factor means software engineering experimentsare vulnerable to subject and experimenter bias. Thelifecycle factor means it is difficult to determine howtechnologies will behave once deployed.Conclusions: Software engineering would benefit fromadopting what it can of the evidence approach providedthat it deals with the specific problems that arise from thenature of software engineering. <s> BIB003 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Software managers and practitioners often must make decisions about what technologies to employ on their projects. They might be aware of problems with their current development practices (for example, production bottlenecks or numerous defect reports from customers) and want to resolve them. Or, they might have read about a new technology and want to take advantage of its promised benefits. However, practitioners can have difficulty making informed decisions about whether to adopt a new technology because there's little objective evidence to confirm its suitability, limits, qualities, costs, and inherent risks. This can lead to poor decisions about technology adoption. Software engineers might make incorrect decisions about adopting new techniques it they don't consider scientific evidence about the techniques' efficacy. They should consider using procedures similar to ones developed for evidence-based medicine. Software companies are often under pressure to adopt immature technologies because of market and management pressures. We suggest that practitioners consider evidence-based software engineering as a mechanism to support and improve their technology adoption decisions. <s> BIB004 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Providing students and professionals with strategies for reliable, cost-effective software development, this guide covers a full spectrum of topics from basic principles and underlying theory to organizational and process issues in real-world application. <s> BIB005 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Open Source Software (OSS) products are widely used although a general consensus on their quality is far to be reached. Providing results on OSS reliability - as quality indicator – contributes to shed some light on this issue and allows organizations to make informed decisions in adopting OSS products or in releasing their own OSS. In this paper, we use a classical technique of Software Reliability Growth to model failures occurrences across versions. We have collected data from the bug tracking systems of three OSS products, Mozilla Firefox, OpenSuse and OpenOffice.org. Our analysis aims at determining and discussing patterns of failure occurrences in the three OSS products to be used to predict reliability behaviour of future releases. Our findings indicate that in the three cases, failures occurrences follow a predetermined pattern, which shows: a) an initial stage in which the community learns the new version b) after this first period a rapid increase of the failure detection rate until c) very few failures are left and the discovery of a new failure discovery is rare. This is the stage in which the version can be considered reliable. <s> BIB006 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Software testing process is an activity, in which the software is verified to comply with the requirements and validated to operate as intended. As software development adopts new development methods, this means also that the test processes need to be changed. In this qualitative study, we observe ten software organizations to understand how organizations develop their test processes and how they adopt new test methods. Based on our observations, organizations do only sporadic test process development, and are conservative when adopting new ideas or testing methods. Organizations need to have a clear concept of what to develop and how to implement the needed changes before they commit to process development. <s> BIB007 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Software testing is an important activity in the software development life-cycle. In an earlier study in 2009, we reported the results of a regional survey of software testing practices among practitioners in the Canadian province of Alberta. To get a larger nationwide view on this topic (across Canada), we conducted a newer survey with a revised list of questions in 2010. Compared to our previous Alberta-wide survey (53 software practitioners), the nation-wide survey had larger number of participants (246 practitioners). We report the survey design, execution and results in this article. The survey results reveal important and interesting findings about software testing practices in Canada. Whenever possible, we also compare the results of this survey to other similar studies, such as the ones conducted in the US, Sweden and Australia, and also two previous Alberta-wide surveys, including our 2009 survey. The results of our survey will be of interest to testing professionals both in Canada and world-wide. It will also benefit researchers in observing the latest trends in software testing industry identifying the areas of strength and weakness, which would then hopefully encourage further industry-academia collaborations in this area. Among the findings are the followings: (1) the importance of testing-related training is increasing, (2) functional and unit testing are two common test types that receive the most attention and efforts spent on them, (3) usage of the mutation testing approach is getting attention among Canadian firms, (4) traditional Test-last Development (TLD) style is still dominating and a few companies are attempting the new development approaches such as Test-Driven Development (TDD), and Behavior-Driven Development (BDD), (5) in terms of the most popular test tools, NUnit and Web application testing tools overtook JUnit and IBM Rational tools, (6) most Canadian companies use a combination of two coverage metrics: decision (branch) and condition coverage, (7) number of passing user acceptance tests and number of defects found per day (week or month) are regarded as the most important quality assurance metrics and decision factors to release, (8) in most Canadian companies, testers are out-numbered by developers, with ratios ranging from 1:2 to 1:5, (9) the majority of Canadian firms spent less than 40% of their efforts (budget and time) on testing during development, and (10) more than 70% of respondents participated in online discussion forums related to testing on a regular basis. <s> BIB008 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Software products quality is strongly influenced by the quality of the process that generated them; particularly, the testing process contributes to product quality and represents a significant effort in software development projects. In this context, this study aim to find which test process models has been defined, adapted or extended in software industry from 1990 to the current date. For this purpose, a systematic literature review has been performed according to relevant guidelines. This study has identified 23 test process models, many of them adapted or extended from TMMi and TPI, which have different architectures and the new ISO/IEC 29119 with an architectural approach aligned to other ISO/IEC software process models. <s> BIB009 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> A software system is context aware when it uses contextual information to help actors (users or other systems) to achieve their tasks. Testing this type of software can be a challenge since context and its variabilities cannot be controlled by the software tester. The ISO/IEC/IEEE 29119 intended to cover testing of any software system. It provides a common language and process for testing software systems, including a categorization of conventional testing techniques. This paper contains the initial results of our ongoing efforts to understand the testing of context aware software, Specifically, we evaluate whether the observed techniques for testing context aware software can be matched against the ISO/IEC/IEEE 29119 categories or if they represent a new breed of testing techniques. The results indicate that using conventional techniques variations to test context aware software systems does not produce evidence on their feasibility to test the context awareness features in such systems. <s> BIB010 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Introduction <s> Mutation Testing has a long research history as a way to improve the quality of software tests. However, it has not yet reached wide consensus for industry-wide adoption, mainly due to missing clear benefits and computational complexity for the application to large systems. In this paper, we investigate the current state of mutation testing support for Java Virtual Machine (JVM) environments. By running an experimental evaluation, we found out that while default configurations are unbearable for larger projects, using strategies such as selective operators, second order mutation and multi-threading can increase the applicability of the approach. However, there is a trade-off in terms of quality of the achieved results of the mutation analysis process that needs to be taken into account. <s> BIB011
Software testing is a key phase of the software development process, as it represents the process of quality validation and verification of a software product BIB005 . Such phase is even more crucial nowadays, as software has become increasingly complex, mission-, safety-critical, and essential in daily activities, calling for an increase in quality BIB002 BIB011 BIB006 . Testing models and standards emerged over time to more clearly define a testing process with formalized criteria to make the whole process repeatable and reproducible BIB010 . Testing standards play a supportive role in reaching the desired quality level, allowing for a more systematic and structured way for further analysis and improvements BIB001 . Over time, several software testing standards and models have emerged to fulfill various requirements of organizations for a testing process  BIB009  . The number of models can allow organizations to adopt the one that mostly satisfies organizational needs, even though full adoption of testing standards by organizations has been found to be sporadic BIB008 BIB007 . The goal of this paper is to identify existing testing models and their characteristics such as domain, context of application, aspects improved, advantages and drawbacks, important models' practices in the industrial context, by complementing existing reviews of Garcia et al. BIB009 , Afzal et al.  , and Garousi et al.  . Information from the review can help practitioners in selecting the most appropriate model for a context. There is a huge variety of methods that can be applied for literature reviews in the context of Evidence Based Software Engineering (EBSE) BIB004 BIB003 . In this paper, we conduct a Systematic Literature Review (SLR)   . The review contributes to the area of testing models by identifying and analyzing the most popular models published in empirical studies. Compared to previous reviews in the area (  BIB009  ), our focus is more on experiences from empirical studies, by providing models characteristics which can be crucial for organizations in the selection process of an appropriate model. In our review, we focus on the following contributions: • the list of the most used testing models applied in empirical studies published in the literature, with the indication of the application domains; • the main aspects supported by the testing models and on which organizations are focused when adopting the model; • the identification of strengths and weaknesses of the testing models and their practices inside organizations as reported by the included cases; The paper is structured as follows. Section 2 explains key terminology of testing standards and results from other existing surveys. Section 3 describes the entire research method and specific steps of the process. Section 4 presents the results of the review answering the research questions. Section 5 discusses the contribution of the review and compare the results with other existing surveys. Section 6 presents possible threats to validity of the review. Section 7 presents the conclusions.