At Amazon, it's a 'hands-off' approach to continuous integration and continuous deployment of software
Amazon has mastered the art and science of automating software deployments, to the point where they don't have to be actively watched -- here's how it's done.
By Joe McKendrick for Service Oriented | June 27, 2020 -- 15:10 GMT (08:10 PDT) | Topic: Enterprise Software
It's no surprise that Amazon Web Services is way ahead of the world with continuous integration and continuous deployment of software, especially since it advertises itself as a go-to place for organizations seeking to put CI/CD into full practice. The online services giant has taken its own internal CI/CD practices to the next level, however, making it essentially a completely "hands-off" operation.
Photo: Joe McKendrick
At AWS, changes in microservices are automatically deployed to production "multiple times a day by continuous deployment pipelines," according to Clare Liguori, a principal software engineer at AWS. This pipeline-centered strategy is key to its ability to keep pumping out code. In a recent post, she explains how Amazon moves software through its phases rapidly and automatically. Remarkably, managers and developers spend little to no time shepherding deployments and watching logs and metrics for any impact. "Automated deployments in the pipeline typically don't have a developer who actively watches each deployment to prod, checks the metrics, and manually rolls back if they see issues. These deployments are completely hands-off. The deployment system actively monitors an alarm to determine if it needs to automatically roll back a deployment."
Software code at AWS moves through four major stages, with automated mechanisms and processes that check and double-check results every step of the way:
Source changes, validated: Amazon's pipelines "automatically validate and safely deploy any type of source change to production, not only changes to application code," says Liguori. "They can validate and deploy changes to sources such as website static assets, tools, tests, infrastructure, configuration, and the application's underlying operation system. All of these changes are version controlled in individual source code repositories. The source code dependencies, such as libraries, programming languages, and parameters like AMI IDs, are automatically upgraded to the latest version at least weekly."
Build, a teamwork process: The code is compiled and unit tested, Liguori continues. "Teams can choose the unit test frameworks, linters, and static analysis tools that work best for them. In addition, teams can choose the configuration of those tools, such as the minimum acceptable code coverage in their unit test framework."
Testing and more testing: The pipeline runs the latest changes through a set of tests and deployment safety checks. "These automated steps prevent customer-impacting defects from reaching production and limit the impact of defects on customers if they do reach production."
Production, incorporating "bake time": Code is released into production staged across AWS regions, to mitigate an issues that arise. "We have found that grouping deployments into 'waves' of increasing size helps us achieve a good balance between deployment risk and speed," Liguori says. "Each wave's stage in the pipeline orchestrates deployments to a group of regions, with changes being promoted from wave to wave. New changes can enter the production phase of the pipeline at any time." She adds that every production rollout has a "bake time" to assess its impact, as "sometimes a negative impact caused by a deployment is not readily apparent. It's slow burning; it doesn't show up immediately during the deployment, especially if the service is under low load at the time. Each prod stage in the pipeline has bake time, which is when the pipeline continues to monitor the team's high-severity aggregate alarm for any slow-burning impact after a deployment is completed and before moving on to the next stage."
AWS just doesn't slap automation onto its processes and hope for the best -- its automated deployment practices are carefully built, tuned and tested, Liguori observes, "based on what helps us balance deployment safety against deployment speed. At the same time, we want to minimize the amount of time developers need to spend worrying about deployments."
Amazon Cloud Big Data Analytics Innovation Tech and Work Collaboration
More from Joe McKendrick
The road to low-code and no-code development needs strong guardrails
3 low-code and no-code trends in the year ahead
IT Priorities
The year ahead in DevOps and agile: Time to instill a sense of urgency
10 most 'disruptive' information technology jobs in the year ahead
How to remove the new "Other Bookmarks" button from Firefox
Mozilla shoves new Firefox button down users' throat via silent update, without an option of disabling it.
Red Hat introduces free RHEL for small production workloads and development teams
In response to complaints about Red Hat's latest plans for CentOS Linux, Red Hat will start offering no-cost Red Hat Enterprise Linux for small production workloads and customer ...
Malwarebytes said it was hacked by the same group who breached SolarWinds
Malwarebytes becomes fourth major security firm targeted by attackers after Microsoft, FireEye, and CrowdStrike.
Survey finds widespread adoption of low-code and no-code approaches. However, IT still needs to be a full partner.
Ethical-source movement opens new open-source organization
Ethical-source leaders are trying to find more support for their open-source approach by creating the Organization for Ethical Source.
Citrix buys Wrike for $2.25 billion, aims to bolster hybrid work portfolio
Citrix also reported its fourth quarter financial results​, which beat expectations.