A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Agile software development is well-known for its focus on close customer collaboration and customer feedback. In emphasizing flexibility, efficiency and speed, agile practices have lead to a paradigm shift in how software is developed. However, while agile practices have succeeded in involving the customer in the development cycle, there is an urgent need to learn from customer usage of software also after delivering and deployment of the software product. The concept of continuous deployment, i.e. the ability to deliver software functionality frequently to customers and subsequently, the ability to continuously learn from real-time customer usage of software, has become attractive to companies realizing the potential in having even shorter feedback loops. However, the transition towards continuous deployment involves a number of barriers. This paper presents a multiple-case study in which we explore barriers associated with the transition towards continuous deployment. Based on interviews at four different software development companies we present key barriers in this transition as well as actions that need to be taken to address these. <s> BIB001 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Scaling a web applications can be easy for simple CRUD software running when you use Platform as a Service Clouds (PaaS). But if you need to deploy a complex software, with many components and a lot users, you will need have a mix of cloud services in PaaS, SaaS and IaaS layers. You will also need knowledge in architecture patterns to make all these software components communicate accordingly. In this article, we share our experience of using cloud services to scale a web application. We show usage examples of load balancing, session sharing, e-mail delivery, asynchronous processing, logs processing, monitoring, continuous deployment, realtime user monitoring (RUM). These are a mixture of development and system operations (DevOps) that improved our application availability, scalability and performance. <s> BIB002 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Rally Software transitioned from shipping code every eight-weeks, with time-boxed Scrum sprints, to a model of continuous delivery with Kanban. The team encountered complex challenges with their build systems, automated test suites, customer enablement, and internal communication. But there was light at the end of the tunnel - greater control and flexibility over feature releases, incremental delivery of value, lower risks, fewer defects, easier on-boarding of new developers, less off-hours work, and a considerable up tick in confidence. This experience report describes the journey to continuous delivery with the aim that others can learn from our mistakes and get their teams deploying more frequently. We will describe and contrast this transition from the business (product management) and engineering perspectives. <s> BIB003 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Abstract Context Continuous Deployment (CD) is an emerging software development process with organisations such as Facebook, Microsoft, and IBM successfully implementing and using the process. The CD process aims to immediately deploy software to customers as soon as new code is developed, and can result in a number of benefits for organisations, such as: new business opportunities, reduced risk for each release, and prevent development of wasted software. There is little academic literature on the challenges organisations face when adopting the CD process, however there are many anecdotal challenges that organisations have voiced on their online blogs. Objective The aim of this research is to examine the challenges faced by organisations when adopting CD as well as the strategies to mitigate these challenges. Method An explorative case study technique that involves in-depth interviews with software practitioners in an organisation that has adopted CD was conducted to identify these challenges. Results This study found a total of 20 technical and social adoption challenges that organisations may face when adopting the CD process. The results are discussed to gain a deeper understanding of the strategies employed by organisations to mitigate the impacts of these challenges. Conclusion While a number of individual technical and social adoption challenges were uncovered by the case study in this research, most challenges were not faced in isolation. The severity of these challenges were reduced by a number of mitigation strategies employed by the case study organisation. It is concluded that organisations need to be well prepared to handle technical and social adoption challenges with their existing expertise, processes and tools before adopting the CD process. For practitioners, knowing how to address the challenges an organisation may face when adopting the CD process provides a level of awareness that they previously may not have had. <s> BIB004 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> As part of a Finnish research program, researchers interviewed 15 information and communications technology companies to determine the extent to which the companies adopted continuous deployment. They also aimed to find out why continuous deployment is considered beneficial and what the obstacles are to its full adoption. The benefits mentioned the most often were the ability to get faster feedback, the ability to deploy more often to keep customers satisfied, and improved quality and productivity. Despite understanding the benefits, none of the companies adopted a fully automatic deployment pipeline. The companies also had higher continuous-deployment capability than what they practiced. In many cases, they consciously chose to not aim for full continuous deployment. Obstacles to full adoption included domain-imposed restrictions, resistance to change, customer desires, and developers' skill and confidence. <s> BIB005 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Continuous delivery and deployment are dramatically shortening release cycles from months to hours. Cloud applications with high-frequency releases often rely heavily on automated tools and cloud infrastructure APIs to deploy new software versions. The authors report on reliability issues and how these tools and APIs contribute to them. They also analyze the trade-offs between using heavily baked and lightly baked virtual-image approaches, on the basis of experiments with Amazon Web Service OpsWorks APIs and the Chef configuration management tool. Finally, they propose error-handling practices for continuous-delivery facilities. <s> BIB006 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> When DevOps started gaining momentum in the software industry, one of the first service-based architectural styles to be introduced, be applied in practice, and become popular was microservices. Migrating monolithic architectures to cloud-native architectures such as microservices reaps many benefits, such as adaptability to technological changes and independent resource management for different system components. This article reports on experiences and lessons learned during incremental migration and architectural refactoring of a commercial MBaaS (mobile back end as a service) to microservices. It explains how adopting DevOps facilitated a smooth migration. Furthermore, the researchers transformed their experiences in different projects into reusable migration practices, resulting in microservices migration patterns. This article is part of a theme issue on DevOps. The Web extra at https://youtu.be/MF3-dKTCQ88 is an audio recording of Brian Brannon speaking with author Pooyan Jamshidi and James Lewis, principal engineer at ThoughtWorks, about DevOps and microservices architecture. <s> BIB007 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Context: Development and Operations (DevOps) is an emerging software industry movement to bridge the gap between software development and operations teams. DevOps supports frequently and reliably releasing new features and products-- thus subsuming Continuous Deployment (CD) practice. Goal: This research aims at empirically exploring the potential impact of CD practice on architecting process. Method: We carried out a case study involving interviews with 16 software practitioners. Results: We have identified (1) a range of recurring architectural challenges (i.e., highly coupled monolithic architecture, team dependencies, and ever-changing operational environments and tools) and (2) five main architectural principles (i.e., small and independent deployment units, not too much focus on reusability, aggregating logs, isolating changes, and testability inside the architecture) that should be considered when an application is (re-) architected for CD practice. This study also supports that software architecture can better support operations if an operations team is engaged at an early stage of software development for taking operational aspects into considerations. Conclusion: These findings provide evidence that software architecture plays a significant role in successfully and efficiently adopting continuous deployment. The findings contribute to establish an evidential body of knowledge about the state of the art of architecting for CD practice <s> BIB008 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Wotif Group used DevOps principles to recover from the downward spiral of manual release activity that many IT departments face. Its approach involved the idea of "making it easy to do the right thing." By defining the right thing (deployment standards) for development and operations teams and making it easy to adopt, Wotif drastically improved the average release cycle time. This article is part of a theme issue on DevOps. <s> BIB009 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Few minutes after a web-based software release, the release team might encounter log traces showing the new system crashing, hanging, or having poor performance. This is the start of the most nerve-wrecking moments of a product's release cycle, i.e., should one run the risk of not doing anything and users losing precious data, or of prematurely engaging the tedious (and costly) roll-back procedure towards the previous release? Thus far, only little attention has been paid by researchers to these so-called "botched releases", partly because of lack of release log data. This paper studies 345 releases of a large e-commerce web app over a period of 1.5 years, in which we identified 17 recurrent root causes of botched releases, classified into four major categories. We then build explanatory models to understand which root causes are the most important, and to explore the factors leading to botched releases. <s> BIB010 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> DevOps is a predominant phenomenon in the web domain. Its two core principles emphasize collaboration between software development and operations, and the use of agile principles to manage deployment environments and their configurations. DevOps techniques, such as collaboration and behaviour-driven monitoring, have been used by web companies to facilitate continuous deployment of new functionality to customers. The techniques may also offer opportunities for continuous product improvement when adopted in the embedded systems domain. However, certain characteristics of embedded software development present obstacles for DevOps adoption, and as yet, there is no empirical evidence of its adoption in the embedded systems domain. In this study, we present the challenges for DevOps adoption in embedded systems using a multiple-case study approach with four companies. The contribution of this paper is to introduce the concept of DevOps adoption in the embedded systems domain and then to identify key challenges for the DevOps adoption. <s> BIB011 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Even though agile actively seeks collaboration from all its stakeholders, most agile projects do not extend themselves toward the operations people. To solve this problem, DevOps is introduced. DevOps is a conceptual framework for reintegrating development and operations of Information Systems, which is able to break the wall between developers and operations professionals. DevOps improves the work through a collection of principles and practices, centered around close collaboration between Development and Operations personnel. However, both sides have paid little attention to issues faced by each other. Communication gaps is a recurrent problem in agile teams that is also eminent in the relationship between developers and operations. Literature offers little research on this aspect of communication in DevOps. This position paper describes the communication practices from a distributed agile team composed of developers and operations based on communication challenges (geographical, socio-cultural, and temporal distance) and strategies (frequency, direction, modality, and content). From the results we outline possible research focus for future work, aiming to enrich the academia research on the matter as well as to help practitioners to improve their working practices. <s> BIB012 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> There are many "continuous" practices in software engineering, for example continuous integration (CI), continuous delivery (CD), continuous release (CR), and DevOps. However, the maintenance aspect of continuity is rarely mentioned in publication or education. The continuous practices and applications depend on many repositories and artifacts, such as databases, servers, virtual machines, storage, data, meta-data, various logs, and reports. Continuous maintenance (CM) seeks to maintain these repositories and artifacts properly and consistently through automation, summarization, compaction, archival, and removal. For example, retaining builds and test results created by CI consumes storage. An automated CM process can remove the irrelevant artifacts and compact the relevant artifacts to reduce storage usage. Proper CM is essential for applications' long term sustainability. There are two sides of CM: pre-production and post-production. During the pre-production phase, CM maintains the health of the development environments and the relevant processes. Then during the post-production phase, CM maintains the health of the applications. This paper defines CM for developers. CM complements and completes continuous practices. <s> BIB013 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> DevOps practices deal with such things as the velocity of releases, how fast incidents are handled, and the enforcement of organizationally specified security practices. All these are critical for success in today’s environment, and the architect is critical for success in adopting DevOps practices. This instalment of the Pragmatic Architect explains why. <s> BIB014 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> It's not magic, it just requires continuous, daily improvement at all levels. <s> BIB015 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Many software services are currently created using DevOps, where developers and operations personnel are more tightly integrated. The DevOps paradigm enables shorter development cycles, but increased speed has raised concerns over whether security issues may be overlooked. However, perfect security is never achievable, and in addition to the proactive software security efforts, we also need a reactive effort to handle flaws and bugs that are not discovered before they are used in an attack. In this paper we explore how focus on incident management and collaboration with developers can contribute to improved software security. <s> BIB016 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> With the emergence of DevOps movement, software engineers are starting to rely on cloud platform APIs for implementing many fault tolerance, self-adaptation and continuous delivery features such as deployment changes, scaling out/in, exception handling, backup/recovery and migration. In this paper we present an empirical study of API issues in commercial cloud platforms. We classified the API failures and their causes, and discuss possible remedies for each category to improve reliability of cloud applications and introduce our solutions to deal with resource-characteristics faults and late timing failures. <s> BIB017 </s> A Survey of DevOps Concepts and Challenges <s> Implications for engineers <s> Defects in infrastructure as code (IaC) scripts can have serious consequences for organizations who adopt DevOps. By identifying which characteristics of IaC scripts correlate with defects, we can identify anti-patterns, and help software practitioners make informed decisions on better development and maintenance of IaC scripts, and increase quality of IaC scripts. The goal of this paper is to help practitioners increase the quality of IaC scripts by identifying characteristics of IaC scripts and IaC development process that correlate with defects, and violate security and privacy objectives. We focus on characteristics of IaC scripts and IaC development that (i) correlate with IaC defects, and (ii) violate security and privacy-related objectives namely, confidentiality, availability, and integrity. For our initial studies, we mined open source version control systems from three organizations: Mozilla, Openstack, and Wikimedia, to identify the defect-related characteristics and conduct our case studies. From our empirical analysis, we identify (i) 14 IaC code and four churn characteristics that correlate with defects; and (ii) 12 process characteristics such as, frequency of changes, and ownership of IaC scripts that correlate with defects. We propose the following studies: (i) identify structural characteristics that correlate with defects; (ii) with respect to prediction performance, compare which characteristics of IaC scripts are more correlated with defects; and (iii) identify characteristics that violate security and privacy objectives. <s> BIB018
Based on the reviewed literature, we list DevOps implications affecting how engineers must architect systems, interact with their peers, and even how to adopt processes, such as incident handling. Microservices: adopting microservices architecture is recommended in conjunction with adopting continuous delivery BIB007 BIB008 . Loosely coupled and well-encapsulated microservices architecture lead to good system testability and deployability BIB015 . Since microservices do not come without new challenges, we further discuss this topic in Section 8.1. Cloud services: architecture patterns involving cloud services can assist in the deployment and operation of applications BIB002 , diminishing the need for dedicated operations teams  . It is particularly useful in the context of cross-functional teams. Rolling back: several authors state that DevOps engineers should be able to roll back applications in case of problems after deployment BIB009 BIB004 . However, in scenarios with complex integration or database evolution BIB009 , rolling back can be a tricky task BIB010 . Alternatives such as feature toggle  or handling the root causes of problematic releases can avoid rolling back  BIB010 . Embedded systems and IoT devices: deployment on embedded systems can be hard, especially when only the customer owns and controls the hardware platform BIB011 . Therefore, engineers must customize the updating mechanisms for embedded systems and IoT devices. Inhibitors for high-frequency delivery: engineers must be aware of scenarios that could impose barriers to high-frequency delivery. Updating the software of automation control systems can require a factory to stop its production BIB005 , which can be costly. Traditional techniques to avoid downtime on deployment, such as Blue Green Deployment [73] and Canary Release [109] , may not be suitable for critical applications, such as for factories, and medical equipment. Another less critical scenario, such as mobile apps, can wait for a release cycle to publish a new update BIB005 . Testing: although companies recognize the importance of automated testing, they still struggle to implement it BIB001 fully. It is especially true for user interface tests automation BIB005 . Other factors that make automated testing complex are hardware availability for load testing and user experiment assessment BIB005 . Parallelizing test suites can be necessary to reduce testing time BIB003 . Tests that seem to fail at random called "flaky tests, " are not acceptable BIB003 . Quality assurance team: quality assurance skills are necessary for locating specific error scenarios and corner cases. However, preserving the quality assurance team separate from the development team is questionable. DevOps and agile practices require a change in the role of quality assurance teams, or even its elimination BIB014 . Legacy systems: although it takes a great deal of work to achieve continuous delivery on mainframe platforms, there are successful reports of it BIB015 . Some legacy architectures might not be designed to run automated tests BIB005 . Nonetheless, teams must be aware that cultural factors, such as managers who say "This is the way we have always done it" BIB015 , can limit the adoption of continuous delivery more than technical factors. Communication: improving communication among organizational silos is key to DevOps adoption  . However, practitioners must be aware that effective interdepartmental communication is still a challenge, especially within remote teams BIB012 . Learning: software professionals must be prepared to learn new tools, focusing on automation  . They must also cope with the ever-changing nature of these tools, which implies maintaining heterogeneous environments and migrating technologies BIB008 . Building the deployment pipeline: the benefits delivered by a deployment pipeline are many. However, engineers must be aware that setting up the infrastructure for continuous deployment can demand a considerable effort BIB005 BIB003 . Breaking down the system into microservices also requires building multiple pipelines. Engineers should not try to build all the continuous delivery ecosystem in a single step: an approach based on continuous improvement is preferable BIB003 . Pipeline maintenance: pipeline execution generates a lot of artifacts, such as build and logs. Artifacts such as production logs, bug tracks, parameters configuration, and temporary files, must be appropriately archived and removed at some point BIB013 . Despite its importance to pipeline sustainability, organizations often overlook this maintenance process BIB013 . On-boarding: the automation built for carrying continuous delivery also promotes faster onboarding of new members to the team BIB003 . Incident handling: software developers must be educated in software security and must cope with incident handling BIB016 , bug tracking, systems failure, or take over primary responsibility for this activity. Coding for stability and security: although software stability and security are traditional operational concerns, in a DevOps context such non-functional requirements must be leveraged by software implementation, including the code for deployment automation and a consideration of the possibility of failures and delays in the underlying infrastructure BIB017 BIB018 BIB006 .