A Survey of DevOps Concepts and Challenges <s> Related Surveys <s> DevOps is a conceptual framework for reintegrating development and operations of Information Systems.We performed a Systematic Mapping Study to explore DevOps. 26 articles out of 139 were selected, studied and summarized. Based on this a concept table was constructed. ::: We discovered that DevOps has not been adequately studied in scientific literature. There is relatively little research available on DevOps and the studies are often of low quality. We also found that DevOps is supported by a culture of collaboration, automation, measurement, information sharing and web service usage. DevOps benefits IS development and operations performance. It also has positive effects on web service development and quality assurance performance. Finally, our mapping study suggests that more research is needed to quantify these effects. <s> BIB001 </s> A Survey of DevOps Concepts and Challenges <s> Related Surveys <s> As the interest in DevOps continues to grow, there is an increasing need for software organizations to understand how to adopt it successfully. This study has as objective to clarify the concept and provide insight into existing challenges of adopting DevOps. First, the existing literature is reviewed. A definition of DevOps is then formed based on the literature by breaking down the concept into its defining characteristics. We interview 13 subjects in a software company adopting DevOps and, finally, we present 11 impediments for the company’s DevOps adoption that were identified based on the interviews. <s> BIB002 </s> A Survey of DevOps Concepts and Challenges <s> Related Surveys <s> Recently, DevOps has emerged as an alternative for software organizations inserted into a dynamic market to handle daily software demands. As claimed, it intends to make the software development and operations teams to work collaboratively. However, it is hard to observe a shared understanding of DevOps, what potentially hinders the discussions in the literature and can confound observations when conducting empirical studies. Therefore, we performed a Multivocal Literature Review aiming at characterizing DevOps in multiple perspectives, including data sources from technical and gray literature. Grounded Theory procedures were used to rigorous analyze the collected data. It allowed us to achieve a grounded definition for DevOps, as well as to identify its recurrent principles, practices, required skills, potential benefits, challenges and what motivates the organizations to adopt it. Finally, we understand the DevOps movement has identified relevant issues in the state-of-the-practice. However, we advocate for the scientific investigations concerning the potential benefits and drawbacks as a consequence of adopting the suggested principles and practices. <s> BIB003 </s> A Survey of DevOps Concepts and Challenges <s> Related Surveys <s> Context: DevOps, the combination of Development and Operations, is a new way of thinking in the software engineering domain that recently received much attention. Given that DevOps is a new term and novel concept recently introduced, no common understanding of what it entails has been achieved yet. Consequently, definitions of DevOps often only represent a part that is relevant to the concept. Objective:This study aims to characterize DevOps by exploring central components of DevOps definitions reported in the literature, specifying practices explicitly proposed for DevOps and investigating the similarities and differences between DevOps and other existing methods in software engineering. Method: A systematic mapping study was conducted that used six electronic databases: IEEE, ACM, Inspec, Scopus, Wiley Online Library and Web of Science. Result: 44 studies have been selected that report a definition of DevOps, 15 studies explicitly stating DevOps practices, and 15 studies stating how DevOps is related to other existing methods. Papers in some cases stated a combination of a definition, practices, and relations to other methods, the total number of primary studies was 49. Conclusion: We proposed a definition for DevOps which may overcome inconsistencies over the various existing definitions of individual research studies. In addition, the practices explicitly proposed for DevOps have been presented as well as the relation to other software development methods. <s> BIB004 </s> A Survey of DevOps Concepts and Challenges <s> Related Surveys <s> Organizations are introducing agile and lean software development techniques in operations to increase the pace of their software development process and to improve the quality of their software. They use the term DevOps, a portmanteau of development and operations, as an umbrella term to describe their efforts. In this paper, we describe the ways in which organizations implement DevOps and the outcomes they experience. We first summarize the results of a systematic literature review that we performed to discover what researchers have written about DevOps. We then describe the results of an exploratory interview-based study involving 6 organizations of various sizes that are active in various industries. As part of our findings, we observed that all organizations were positive about their experiences and only minor problems were encountered while adopting DevOps. <s> BIB005
Even though DevOps is a recent research topic, several studies have performed a Systematic Literature Review (SLR) on the subject. They aim at finding a final DevOps definition and which aspects influence its adoption on an organization. Table 2 depicts the major research questions of these previous SLR. It is worth to mention most previous SLRs performed their literature review before 2015. It means they primarily examined the pioneering works on DevOps and did not capture the significant growth of publications depicted in Figure 2 ; therefore, they miss most of the currently existing literature in the topic. This is evidenced by their conclusion that initial studies have a very low quality BIB005 . BIB003 BIB005  BIB002 The most relevant contribution of these previous studies is to correlate the absence of a wellestablished DevOps definition with its effect on the perception of its benefits, expectations, and adoption challenges BIB005 . To overcame the lack of the initial studies quality, they deployed methods other than Systematic Literature Review to uncover both academic and practitioners perspectives and to address the novelty of DevOps. Table 3 presents a list of sources used by these other works. Our survey selected works from all cited sources and had a much broader range of analyzed works than previous SLRs. We identified 167 papers and carefully examined their titles and abstracts to select the 50 core papers, which were then extensively analyzed to provide an updated overview of DevOps. Erich et al. BIB001 BIB005 is the most referenced literature review of DevOps. The authors performed an SLR BIB001 BIB005 and conducted focused interviews on DevOps principles and practices at six organizations BIB005 . From SLR, they extracted seven areas related to DevOps: "culture of collaboration, automation, measurement, sharing, services, quality assurance, and governance" BIB005 . They conclude that there was, in general, low quality on the academic studies on DevOps and a lack of studies to evidence DevOps' effectiveness. From focused interviews, they analyzed how organizations implemented DevOps in four dimensions: governance, personal traits, department, and practitioners. They categorized answers from each organization into one of the seven areas extracted from SLR to portray their perceptions. Although both academic studies and organizations stated the benefits of adopting DevOps, Erich et al. concluded that no quantitative studies support such a claim BIB005 , and reinforce the necessity of experimental studies to verify the effectiveness of DevOps. França et al. BIB003 assumed academic studies were insufficient to address DevOps thoroughly, so they alternatively conducted a Multivocal Literature Review (MLR), in which most of the sources were extracted from gray literature, including books, websites, and industry journals and technical reports. The authors describe DevOps as a collection of principles, practices, required skills, and organizations' motivations to adopt it. They identified seven areas related to DevOps: social aspects, automation, measurements, sharing, quality assurance, and leanness. Their main contribution was to describe DevOps characteristics associated with the practitioners' community and state-of-thepractice. Erich et al. BIB005 and França et al. BIB003 indicate the need for more research on DevOps, especially quantitative research to evaluate DevOps' effectiveness. Even with distinct methodologies, these two works arrived at similar conclusions. They concentrate their studies on the Process and People categories of our conceptual map, to be described in Section 5. Previous surveys BIB003  BIB004 did not cover the Runtime and Delivery categories of our conceptual map; they have only mentioned some of the concepts. None of the previous SLRs have discussed the technical implications and complexity of adopting DevOps practices such as automation, microservices architectures, containerization, and toolset management. The concepts and implications of practical aspects of DevOps not only empower managers to make assertive and strategic decisions but also equip engineers with best practices for DevOps adoption. We provide lessons learned of i) using DevOps with legacy systems; ii) the complexities and mistakes when adopting the microservice architecture; iii) guidelines for automating the delivery pipeline; iv) the advantages and drawbacks of teams autonomy in toolset selection. These points form the major contribution of our survey, wherein we deepen the analysis of practical aspects of DevOps adoption with the aid of a conceptual map of DevOps to guide engineers, managers, and researchers toward mastering skills necessary to enable DevOps. Additionally, the quality of studies on DevOps improved in recent years, so we have updated the implications of deploying DevOps in an organization as well as contributed to more technical implications for practitioners and researchers, such as re-designing systems architecture, deployment pipeline, and quantitative assessment metrics. Previous SLRs did not cover these latest subjects.