A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Selection of online-digital libraries <s> Systematic Literature Reviews and Systematic Mapping Studies are relatively new forms of secondary studies in software engineering. Identifying relevant papers from various Electronic Data Sources (EDS) is one of the key activities of conducting these kinds of studies. Hence, the selection of EDS for searching the potentially relevant papers is an important decision, which can affect a study's coverage of relevant papers. Researchers usually select EDS mainly based on personal knowledge, experience, and preferences and/or recommendations by other researchers. We believe that building an evidence-based understanding of EDS can enable researchers to make more informed decisions about the selection of EDS. This paper reports our initial effort towards this end. We propose an initial set of metrics for characterizing the EDS from the perspective of the needs of secondary studies. We explain the usage and benefits of the proposed metrics using the data gathered from two secondary studies. We also tried to synthesize the data from the two studies and that from literature to provide initial evidence-based heuristics for EDS selection. <s> BIB001 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Selection of online-digital libraries <s> Open innovation (OI) means that innovation is fostered by using both external and internal influences in the innovation process. In software engineering (SE), OI has existed for decades, while we currently see a faster and broader move towards OI in SE. We therefore survey research on how OI takes place and contributes to innovation in SE. This study aims to synthesize the research knowledge on OI in the SE domain. We launched a systematic mapping study and conducted a thematic analysis of the results. Moreover, we analyzed the strength of the evidence in the light of a rigor and relevance assessment of the research. We identified 33 publications, divided into 9 themes related to OI. 17/33 studies fall in the high---rigor/high---relevance category, suggesting the results are highly industry relevant. The research indicates that start-ups have higher tendency to opt for OI compared to incumbents. The evidence also suggests that firms assimilating knowledge into their internal R&D activities, have higher likelihood of gaining financial advantages. We concluded that OI should be adopted as a complementary approach to facilitate internal innovation and not to substitute it. Further research is advised on situated OI strategies and the interplay between OI and agile practices. <s> BIB002 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Selection of online-digital libraries <s> Abstract Context Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines. Objective To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly. Method We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment). Results In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given. Conclusion The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings. <s> BIB003 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Selection of online-digital libraries <s> ContextAny newcomer or industrial practitioner is likely to experience difficulties in digesting large volumes of knowledge in software testing. In an ideal world, all knowledge used in industry, education and research should be based on high-quality evidence. Since no decision should be made based on a single study, secondary studies become essential in presenting the evidence. According to our search, over 101 secondary studies have been published in the area of software testing since 1994. With this high number of secondary studies, it is important to conduct a review in this area to provide an overview of the research landscape in this area. ObjectiveThe goal of this study is to systematically map (classify) the secondary studies in software testing. We propose that tertiary studies can serve as summarizing indexes which facilitate finding the most relevant information from secondary studies and thus supporting evidence-based decision making in any given area of software engineering. Our research questions (RQs) investigate: (1) Software-testing-specific areas, (2) Types of RQs investigated, (3) Numbers and Trends, and (4) Citations of the secondary studies. MethodTo conduct the tertiary study, we use the systematic-mapping approach. Additionally, we contrast the testing topics to the number of Google hits to address a general popularity of a testing topic and study the most popular papers in terms of citations. We furthermore demonstrate the practicality and usefulness of our results by mapping them to ISTQB foundation syllabus and to SWEBOK to provide implications for practitioners, testing educators, and researchers. ResultsAfter a systematic search and voting process, our study pool included 101 secondary studies in the area of software testing between 1994 and 2015. Among our results are the following: (1) In terms of number of secondary studies, model-based approach is the most popular testing method, web services are the most popular system under test (SUT), while regression testing is the most popular testing phase; (2) The quality of secondary studies, as measured by a criteria set established in the community, is slowly increasing as the years go by; and (3) Analysis of research questions, raised and studied in the pool of secondary studies, showed that there is a lack of 'causality' and 'relationship' type of research questions, a situation which needs to be improved if we, as a community, want to advance as a scientific field. (4) Among secondary studies, we found that regular surveys receive significantly more citations than SMs (p=0.009) and SLRs (p=0.014). ConclusionDespite the large number of secondary studies, we found that many important areas of software testing currently lack secondary studies, e.g., test management, role of product risk in testing, human factors in software testing, beta-testing (A/B-testing), exploratory testing, testability, test stopping criteria, and test-environment development. Having secondary studies in those areas is important for satisfying industrial and educational needs in software testing. On the other hand, education material of ISTQB foundation syllabus and SWEBOK could benefit from the inclusion of the latest research topics, namely search-based testing, use of cloud-computing for testing and symbolic execution. <s> BIB004 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Selection of online-digital libraries <s> Context: A number of Systematic Mapping Studies (SMSs) that cover Software Engineering (SE) are reported in literature. Tertiary studies synthesize the secondary studies to provide a holistic view of an area. Objectives: We synthesize SMSs in SE to provide insights into existing SE areas and to investigate the trends and quality of SMSs. Methodology: We use Systematic Literature Review protocol to analyze and map the SMSs in SE, till August 2017, to SE Body of Knowledge (SWEBOK). Results: We analyze 210 SMSs and results show that: (1) Software design and construction are most active areas in SE; (2) Some areas lack SMSs, including mathematical foundations, software configuration management, and SE tools; (3) The quality of SMSs is improving with time; (4) SMSs in journals have higher quality than SMSs in conferences and are cited more often; (5) Low quality in SMSs can be attributed to a lack of quality assessment in SMSs and not reporting information about the primary studies. Conclusion: There is a potential for more SMSs in some SE areas. A number of SMSs do not provide the required information for an SMS, which leads to a low quality score. <s> BIB005
A search for the relevant articles was carried out to answer the research questions. We focused on major digital libraries (given in Table 2 ) and augmented the search process using a welldefined methodology of snowballing used by other studies BIB005 BIB002 BIB003 BIB004 . For snowballing we follow the guidelines by Wohlin et al.  . Suitable repositories were identified based on previous research experience and suggestions provided by Chen et al. BIB001 . The automated search process resulted in a number of duplicate studies in the initial search but we preferred a conservative approach over reducing redundancy in the initial search results. The search query formulation is discussed in subsection 2.1.4. 