A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> There is a documented gap between academic and practitioner views on software testing. This paper tries to close the gap by investigating both views regarding the benefits and limits of test automation. The academic views are studied with a systematic literature review while the practitioners views are assessed with a survey, where we received responses from 115 software professionals. The results of the systematic literature review show that the source of evidence regarding benefits and limitations is quite shallow as only 25 papers provide the evidence. Furthermore, it was found that benefits often originated from stronger sources of evidence (experiments and case studies), while limitations often originated from experience reports. We believe that this is caused by publication bias of positive results. The survey showed that benefits of test automation were related to test reusability, repeatability, test coverage and effort saved in test executions. The limitations were high initial invests in automation setup, tool selection and training. Additionally, 45% of the respondents agreed that available tools in the market offer a poor fit for their needs. Finally, it was found that 80% of the practitioners disagreed with the vision that automated testing would fully replace manual testing. <s> BIB001 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> Context: The web has had a significant impact on all aspects of our society. As our society relies more and more on the web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 193 papers in the area of web application testing, which have appeared between 2000 and 2013. Objective: As this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends and empirical evidence in this specialized field. Methods: We systematically review the body of knowledge related to functional testing of web application through a systematic literature review (SLR) study. This SLR is a follow-up and complimentary study to a recent systematic mapping (SM) study that we conducted in this area. As part of this study, we pose three sets of research questions, define selection and exclusion criteria, and synthesize the empirical evidence in this area. Results: Our pool of studies includes a set of 95 papers (from the 193 retrieved papers) published in the area of web application testing between 2000 and 2013. The data extracted during our SLR study is available through a publicly-accessible online repository. Among our results are the followings: (1) the list of test tools in this area and their capabilities, (2) the types of test models and fault models proposed in this domain, (3) the way the empirical studies in this area have been designed and reported, and (4) the state of empirical evidence and industrial relevance. Conclusion: We discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our SLR can help researchers to obtain an overview of existing web application testing approaches, fault models, tools, metrics and empirical evidence, and subsequently identify areas in the field that require more attention from the research community. <s> BIB002 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> Time and resource constraints should be taken into account in software testing activities, and thus optimizing the test suite is fundamental in the development process. In this context, the test case selection aims to eliminate redundant or unnecessary test data, which is crucial for the definition of test strategies. This paper presents a systematic review on the test case selection conducted through a selection of 449 articles published in leading journals and conferences in Computer Science. We addressed the state-of-art by collecting and comparing existing evidence on the methods used in the different software domains and the methods used to evaluate the test case selection. Our study identified 32 papers that met the research objectives, which featured 18 different selection methods and were evaluated through 71 case studies. The most commonly reported methods are adaptive random testing, genetic algorithms and greedy algorithm. Most approaches rely on heuristics, such as diversity of test cases and code or model coverage. This paper also discusses the key concepts and approaches, areas of application and evaluation metrics inherent to the methods of test case selection available in the literature. <s> BIB003 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> Context: Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but techniques that can take advantage of these opportunities are still needed. Objective: The objective of this study is to identify testing strategies that have the potential to achieve these economies, and to provide a synthesis of available research on SPL testing strategies, to be applied towards reaching higher defect detection rates and reduced quality assurance effort. Method: We performed a literature review of two hundred seventy-six studies published from the year 1998 up to the 1st semester of 2013. We used several filters to focus the review on the most relevant studies and we give detailed analyses of the core set of studies. Results: The analysis of the reported strategies comprised two fundamental aspects for software product line testing: the selection of products for testing, and the actual test of products. Our findings indicate that the literature offers a large number of techniques to cope with such aspects. However, there is a lack of reports on realistic industrial experiences, which limits the inferences that can be drawn. Conclusion: This study showed a number of leveraged strategies that can support both the selection of products, and the actual testing of products. Future research should also benefit from the problems and advantages identified in this study. <s> BIB004 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. Results: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. Conclusions: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques. <s> BIB005 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> a. SLRs in Software Testing <s> Abstract Context Empirical studies are essential in evaluating the effectiveness of Model-based Testing (MBT) research and should be reported properly to ensure their replication and to highlight the strengths and limitations of the MBT techniques being evaluated. Researchers have proposed guidelines detailing what information should be reported when presenting empirical studies and what should be the structure of such primary studies. There is a need to evaluate the reporting quality of the empirical studies in MBT literature. Objective To evaluate the reporting quality of empirical studies in the model based testing domain; identifying where the reported studies fail to follow the proposed guidelines and finding frequently omitted details. As an auxiliary goal we aim to quantify the percentage of empirical studies conducted in industrial context. Method We evaluate the reporting quality and the execution contexts of MBT empirical studies reported in literature. For our study we consider the MBT papers published in top ten software engineering journals over the last eighteen years. We evaluate the published primary studies using the empirical study reporting guidelines. Results We found 87 empirical in MBT that met our selection criteria. Initial results showed that the existing guidelines were not only too strict (for example they demand presence of specific sections rather than simply having the details present in the paper), they also did not adequately cover MBT specific details. Therefore, we propose modified the guidelines for reporting empirical studies in MBT and re-evaluated the selected studies. Results show that while only a few empirical studies follow the exact structure proposed by the guidelines, approximately half the papers contain at least 50% of the required details. Most of the papers omit details related to process and analysis leading to presented results. We found a positive trend of improving reporting quality of empirical studies in MBT over the last Eighteen years. Another important finding from the review is that few reported studies were conducted in real industrial context. Conclusions Model based testing community needs to be more aware of the reporting guidelines and more effort should be spent on reporting the necessary details. Furthermore, we found that only few studies that are conducted in industrial context and hence more focus should be given to empirical case studies in real industry context. However, the reporting quality of research papers presenting empirical evaluations is gradually improving. <s> BIB006
There are a number of systematic reviews in different sub-areas of software testing  . For example, Dogan et al. BIB002 conducted an SLR on web application testing to identify, analyze and classify state-of-the-art techniques for testing of web applications. Kanewaka et al. BIB005 systematically gathered literature on the challenges and proposed solutions to testing of scientific software. Catal et al.  presented a systematic literature review on test case prioritization techniques using a genetic algorithm. The paper summarizes the existing techniques of the genetic algorithm for test case prioritization. Narciso et al. BIB003 conducted an SLR on the techniques of test case selection and state that random testing, genetic algorithm and greedy algorithm are the most commonly reported methods. Machado et al. BIB004 presented a systematic review on the strategies used in testing of software product lines. Rafi et al. BIB001 summarize the benefits and limitations of automated software testing by analyzing papers that presented techniques for test automation. Khan et al. BIB006 conducted an SLR on the reporting quality of model-based testing techniques.