Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Search strategy <s> In the decade since the idea of adapting the evidence-based paradigm for software engineering was first proposed, it has become a major tool of empirical software engineering. Evidence-Based Software Engineering and Systematic Reviews provides a clear introduction to the use of an evidence-based model for software engineering research and practice. The book explains the roles of primary studies (experiments, surveys, case studies) as elements of an over-arching evidence model, rather than as disjointed elements in the empirical spectrum. Supplying readers with a clear understanding of empirical software engineering best practices, it provides up-to-date guidance on how to conduct secondary studies in software engineeringreplacing the existing 2004 and 2007 technical reports. The book is divided into three parts. The first part discusses the nature of evidence and the evidence-based practices centered on a systematic review, both in general and as applying to software engineering. The second part examines the different elements that provide inputs to a systematic review (usually considered as forming a secondary study), especially the main forms of primary empirical study currently used in software engineering. The final part provides practical guidance on how to conduct systematic reviews (the guidelines), drawing together accumulated experiences to guide researchers and students in planning and conducting their own studies. The book includes an extensive glossary and an appendix that provides a catalogue of reviews that may be useful for practice and teaching. <s> BIB001 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Search strategy <s> Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to a diversity of software and organisations. <s> BIB002 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Search strategy <s> The systematic evaluation of ideas by experiments are the foundation of continuous experimentation. It allows to assess the value of an idea, remove guessing and subjective opinions from the discussion. The enormous interest of it by practitioners and researchers let the body of knowledge consistently grow. New framework, methods and techniques are developed and its application is constantly expanded to new fields like cyber-physical systems or social networks. In this paper we present a systematic mapping study to characterize the current state of research on continuous experimentation. Our study analyzes the following aspects: intensity of research activity and industry-academia collaboration, influential authors and publications, frequent research types and topics, kind of contributions and terms used for continuous experimentation. Our findings show amongst others that the intensity of research activities increases consistently, the collaboration between industry and academia is high and the most cited publications are experience reports from practitioners. <s> BIB003 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Search strategy <s> Context: Continuous experimentation is frequently used in web-facing companies and it is starting to gain the attention of embedded systems companies. However, embedded systems companies have different challenges and requirements to run experiments in their systems. Objective: This paper explores the challenges during the adoption of continuous experimentation in embedded systems from both industry practice and academic research. It presents strategies, guidelines, and solutions to overcome each of the identified challenges. Method: This research was conducted in two parts. The first part is a literature review with the aim to analyze the challenges in adopting continuous experimentation from the research perspective. The second part is a multiple case study based on interviews and workshop sessions with five companies to understand the challenges from the industry perspective and how they are working to overcome them. Results: This study found a set of twelve challenges divided into three areas; technical, business, and organizational challenges and strategies grouped into three categories, architecture, data handling and development processes. Conclusions: The set of identified challenges are presented with a set of strategies, guidelines, and solutions. To the knowledge of the authors, this paper is the first to provide an extensive list of challenges and strategies for continuous experimentation in embedded systems. Moreover, this research points out open challenges and the need for new tools and novel solutions for the further development of experimentation in embedded systems. <s> BIB004
The search string was initially based on relevant related works that explored the literature with the aim of covering what progress has been made about the general study and adoption of Continuous Experimentation BIB002 BIB003 . As our goal is to focus on the adoption of Continuous Experimentation in cyber-physical systems, in the example of the automotive industry, we added to the search string relevant terms that would steer the scope of the search to our needs. Due to the novelty of the Continuous Experimentation practice and the lack of a globally accepted name in all the sub-disciplines that adopt this practice or variations thereof, many synonyms were needed in the search string in order to aim for inclusiveness. The majority of these search terms were used also by those related works that run comprehensive literature explorations. The variation problem does not appear for the cyber-physical systems field, which is a more established research context with a widely accepted terminology. The final string is thus: The search string was queried on the following databases: ACM Digital Library, IEEE Xplore, Scopus, and Web of Science, returning a total of 192 publications (results up do date as of October 2019). To improve the completeness of the search results, as suggested by BIB001 , a set of 12 papers were used as the basis for a manual backwards snowballing phase, which added 211 publications. The papers were chosen among the works included in past literature explorations BIB002 BIB003 BIB004 based on how close their scope and focus was to this work. Successively the duplicates removal phase took place. All results from the database and snowball search were collected in CSV format and a script comparing entries by publication title removed works which appeared more than once. 