Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> Business and design decisions regarding software development should be based on data, not opinions among developers, domain experts or managers. The company running the most and fastest experiments among the customer base against the lowest cost per experiment outcompetes others by having the data to engineer products with outstanding qualities such as power consumption and user experience. Innovation experiment systems for mass-produced devices with embedded software is an evolution of current RaD practices, going from where innovations are internally evaluated by the original equipment manufacturer to where they are tried by real users in a scale relevant to the full customer base. The turnaround time from developing and deploying an embedded product to getting customer feedback is decreased to weeks, the limit being the speed of the software development teams. The paper presents an embedded architecture for realising such a novel innovation experiment system based on a set of scenarios of what to evaluate in the experiments. A case is presented implementing an architecture in a prototype in-vehicle infotainment system where comparative testing between two software alternatives was performed. <s> BIB001 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> The paper discusses the concept of innovation experiment systems in the context of long-lived embedded systems. These systems need to evolve continuously to stay competitive and provide value to the customer and end-user, especially in domains where the pace of change is increasing. ::: ::: Innovation experiment systems provide a natural mechanism that allows an embedded system, its architecture and underlying platform to continuously evolve in response to changes in the user requirements and system context. It uses a rapid feedback loop to evaluate the benefits of small variations to users with the intent of continuous improvements. ::: ::: The paper explores the architectural implications as the ability to continuously evolve and conduct experiences in the deployed product context in a safe and controlled manner must be supported by the architecture of the embedded systems. ::: ::: Finally, the paper illustrates these concepts using a case study concerning an infotainment system in the automotive industry. <s> BIB002 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> Traditional software development focuses on specifying and freezing requirements early in the, typically yearly, product development lifecycle. The requirements are defined based on product management's best understanding. The adoption of SaaS and cloud computing has shown a different approach to managing requirements, adding frequent and rigorous experimentation to the development process with the intent of minimizing R&D investment between customer proof points. This offers several benefits including increased customer satisfaction, improved and quantified business goals and the transformation to a continuous rather than waterfall development process. In this paper, we present our learnings from studying software companies applying an innovation experiment system approach to product development. The approach is illustrated with three cases from Intuit, the case study company. <s> BIB003 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> The systematic evaluation of ideas by experiments are the foundation of continuous experimentation. It allows to assess the value of an idea, remove guessing and subjective opinions from the discussion. The enormous interest of it by practitioners and researchers let the body of knowledge consistently grow. New framework, methods and techniques are developed and its application is constantly expanded to new fields like cyber-physical systems or social networks. In this paper we present a systematic mapping study to characterize the current state of research on continuous experimentation. Our study analyzes the following aspects: intensity of research activity and industry-academia collaboration, influential authors and publications, frequent research types and topics, kind of contributions and terms used for continuous experimentation. Our findings show amongst others that the intensity of research activities increases consistently, the collaboration between industry and academia is high and the most cited publications are experience reports from practitioners. <s> BIB004 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> Context: Continuous experimentation is frequently used in web-facing companies and it is starting to gain the attention of embedded systems companies. However, embedded systems companies have different challenges and requirements to run experiments in their systems. Objective: This paper explores the challenges during the adoption of continuous experimentation in embedded systems from both industry practice and academic research. It presents strategies, guidelines, and solutions to overcome each of the identified challenges. Method: This research was conducted in two parts. The first part is a literature review with the aim to analyze the challenges in adopting continuous experimentation from the research perspective. The second part is a multiple case study based on interviews and workshop sessions with five companies to understand the challenges from the industry perspective and how they are working to overcome them. Results: This study found a set of twelve challenges divided into three areas; technical, business, and organizational challenges and strategies grouped into three categories, architecture, data handling and development processes. Conclusions: The set of identified challenges are presented with a set of strategies, guidelines, and solutions. To the knowledge of the authors, this paper is the first to provide an extensive list of challenges and strategies for continuous experimentation in embedded systems. Moreover, this research points out open challenges and the need for new tools and novel solutions for the further development of experimentation in embedded systems. <s> BIB005 </s> Continuous Experimentation and the Cyber-Physical Systems challenge. An overview in literature and the industrial perspective <s> Quality assurance <s> Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to a diversity of software and organisations. <s> BIB006
• Performance. Running various instances of the software can be very demanding to the automotive hardware, which is typically resource-constrained. • Remote execution. The risk of unwanted or unknown behavior of the system is increased. Moreover, updates could be at risk of not occurring due to poor, faulty, or non-existing network connections. • Testing. At this point in time the Continuous Experimentation practice has been recently studied in literature as noted also in BIB004 , although in the context of cyber-physical systems this has happened with a quite limited number of strategies and studies. As shown in the presented systematic literature review, the majority of studies have a high-level approach. This means that they try to tackle from a more conceptual point of view the difficulty of applying this practice to a new field which faces different challenges than the field from which Continuous Experimentation originates. Many of these studies are observational, in which a case study is run to analyze whether a certain hypothesis is met in practice BIB005   BIB001 BIB002 BIB003 . A minority of articles are instead design studies trying to draft possible solutions to the technical hurdles opposing the adoption of Continuous Experimentation on cyber-physical systems BIB001 . This unbalance towards more theoretical studies is assumed by the authors to be a direct effect of the relative novelty of the practice in object in the field of cyber-physical systems: in time the authors expect to see an increase in more technical studies facing and overcoming the challenges identified in this more investigative initial period. Drawing a comparison between these results and the ones reported in related studies, both BIB006 and BIB004 Comparing the previous multiple case study with the current one, some items did not emerge in the latest cases, e.g., the presence of a fallback plan in case of failures during the experimentation process, or the risks associated with needing to exchange data with a product in an area where it cannot establish a successful connection, or the challenge to manage heterogeneous hardware configuration in different product families. The overlap between the Challenges items in the two multiple case studies shows to be higher than what was seen in the Advantages list, meaning perhaps that more agreement is found when discussing obstacles to the adoption of Continuous Experimentation in this field.