A Survey of DevOps Concepts and Challenges <s> How to assess the quality of DevOps practices in organizations <s> As part of a Finnish research program, researchers interviewed 15 information and communications technology companies to determine the extent to which the companies adopted continuous deployment. They also aimed to find out why continuous deployment is considered beneficial and what the obstacles are to its full adoption. The benefits mentioned the most often were the ability to get faster feedback, the ability to deploy more often to keep customers satisfied, and improved quality and productivity. Despite understanding the benefits, none of the companies adopted a fully automatic deployment pipeline. The companies also had higher continuous-deployment capability than what they practiced. In many cases, they consciously chose to not aim for full continuous deployment. Obstacles to full adoption included domain-imposed restrictions, resistance to change, customer desires, and developers' skill and confidence. <s> BIB001 </s> A Survey of DevOps Concepts and Challenges <s> How to assess the quality of DevOps practices in organizations <s> Your biggest mistake might be collecting the wrong data. <s> BIB002 </s> A Survey of DevOps Concepts and Challenges <s> How to assess the quality of DevOps practices in organizations <s> Over the past three years, Fannie Mae IT has transformed from a traditional waterfall organization to a lean culture enabled by agile methods and DevOps. Software analytics were used to guide improvements and evaluate progress. Project-level analytics enabled agile teams to improve structural quality and evaluate their practices as they delivered greater functionality over shrinking delivery intervals. Aggregated enterprise metrics displayed an average 38 percent improvement in structural quality, with some applications achieving 48 percent to 70 percent gains. These quality improvements accompanied a 28 percent increase in productivity. The frequency of releases shrank from cycles of nine to 18 months, down to releases every couple of one- to two-month sprints. The article discusses examples of how analytics were used, along with challenges in selecting measures and implementing analytics in an agile–DevOps transformation. This article is part of a special issue on Actionable Analytics for Software Engineering. <s> BIB003 </s> A Survey of DevOps Concepts and Challenges <s> How to assess the quality of DevOps practices in organizations <s> Software producing organizations aim to release high quality software faster, which triggers the adoption of DevOps. However, not many artifacts are available that aid in adopting DevOps. In an attempt to bridge this gap, a DevOps Competence Model showing an overview of the areas to be considered in adopting DevOps is proposed. Also, a DevOps Maturity Model is proposed that presents a growth path for software producing organizations. Both these models incorporate perspectives that are made up of focus areas which in turn are made up of capabilities. Apart from designing and validating these models by means of expert workshops, a case study has been conducted where assessees answered questions to gain insight into which capabilities were implemented. From the answers, maturity profiles were extracted that supported the assessees in becoming more DevOps mature. <s> BIB004
Some researchers have proposed maturity models for DevOps adoption and continuous delivery adoption BIB001 51,  . Feijter et al., for example, provide maturity DevOps models to help organizations measure their current maturity level and to determine areas that require additional investment BIB004 . However, none of these maturity models are widely adopted by industry. Indeed, most organizations have almost no visibility or reliable measurement of their software-delivery practices BIB002 . Forsgren and Kersten claim that both survey data and system data BIB002 should measure DevOps transformations. System data can provide a continuous flow of information, although setting up the aggregation of metrics from different sources can be challenging. Survey data provides a holistic view of out-of-the system issues, such as culture, job satisfaction, and burnout, and can even point to problems on system-data collection. Whether survey or system data, Forsgren and Kersten still make clear that if results are used to punish teams, data collection will be unreliable BIB002 , as also reinforced by Brown from Microsoft  . Kua cautions that inappropriate use of metrics can lead to undesired behavior and even divert the organization from its goals  . Kua also compiles guidelines for better usage of metrics by: explicitly linking metrics to goals, favoring trends over absolute numbers, tracking shorter periods, and changing metrics when they do not drive change anymore. Some examples of metrics based on survey data used on the State of DevOps Reports [53,  are: IT performance as a function of time from commit to deployment, frequency of deployment, and recovery time. Time spent on unplanned work and rework. Typology of organizational culture (pathological, bureaucratic or generative) based on climate for learning, job satisfaction, developers and operators having win-win relationships, usage of version control, automated testing. Employee engagement, based on the Net Promoter Score, indicating how likely employees are to recommend the company products and services. Snyder and Curtis use metrics to evaluate DevOps in a specific company BIB003 . Metrics such as productivity, defect ratio, dollars spent, cycle time release, and build count were aggregated in a "total quality index". More importantly, the aggregation of data from different silos was used to align organizational efforts. The authors observed a higher increase in the total quality index of agile and DevOps teams, which helped executives justify investments in the agile-DevOps transformation. There is a severe lack of industry consensus on how to measure software delivery BIB002 , and designing a good survey requires some expertise. Thus, researchers should provide standards for collecting and analyzing evaluation metrics for the software delivery process, as done by Feijter et al. BIB004 . In the health sector, this model is mature, wherein researchers instrument therapists with standard protocols for diagnoses purposes. Nonetheless, Forsgren and Kersten provide some general guidelines for survey application: it should be limited to 20 to 25 minutes and applied every four to six months BIB002 . Forsgren and Kersten also advise avoiding surveying management and executives, since they tend to overestimate the maturity of their organizations. Researchers can also run the surveys across multiple organizations as done in the State of DevOps Reports. The results of these questionnaires can help practitioners in orienting their careers and help organizations to self-evaluate their performance by comparing themselves to their peers.