How did we get here?

It's 8 a.m. on a Saturday and the release party's post-mortem has finally been completed. Throughout the release, every encountered issue resulted in a Root Cause Analysis process. Once each of the RCAs were done, the release team would then create and assign tickets as needed, resulting in action items for the different teams in the Engineering organization. With the post-mortem being completed, the release team can hand-off the monitoring of the production application to the weekend support team and head home.

The final production servers were upgraded with the new application release at around 3 a.m. that morning with all of the application health checks successfully passing by 3:30 a.m. And yet, even with the early morning finishing time, this is a significant improvement when comparing it to the release parties of a few years ago. Previously, the applications were released every 6 to 12 months, rather than the quarterly release cadence that the company is currently on.

Their company had hired a consulting agency to advise them on how to improve their application's mean-time-to-market and reduce their production outages in order to meet business initiatives and demands. The outcome suggested by this consulting agency was to release the application more frequently than once or twice a year. As a result, the releases have been quicker and less prone to error, which the business has taken notice of. The release parties still require pulling an all-nighter, but the previous release parties were more like all-weekenders or longer.

The on-call engineering team still has to be brought in for every release, but at least they aren't required to be a part of the release party for the entire time. And the most recent release only required a conference bridge for about 4 hours to solve issues with the underlying code or provide quick fixes. Overall, the operations team, infrastructure team, network team, and security team were able to solve most of the issues that showed up, which accounted for significantly more confidence in the newer release cadence.

The different teams should be able to accomplish the backlog of issues before the next release. But the team with the largest issue backlog were the systems administrators , who build, integrate, administrate, and troubleshoot the many different tools used during the releases.

After 12 straight hours with over 15 members across a host of different teams, the release party was complete. When considering the time associated with the attempt to improve the process throughout the quarter, as well as the actual release itself, it is not difficult to run the mental math on the associated costs. The teams need to figure out a way to make the releases more reliable, repeatable, and scalable.

This analogy is all too familiar for many who have been involved in the engineering side of a business during the Waterfall software development life cycle days. When applications were first made available as a SaaS (Software as a Service) solution,the common release cadence was an annual release. Throughout the year a company would deploy small release, often called patches, which mainly consisted of hardware, software, or security updates.

Since the yearly update was essentially releasing a brand-new product, the release process required significant involvement from every team across the entire engineering organization. The release was a major event , often taking an entire weekend, or longer, from every team available. Many in the industry had dubbed this event a release party. Each release party included significant amounts of caffeine and food, which accompanied a host of people hunched over their laptops as they watched the output of the release on a massive projector screen. Yet the worst part of this whole scenario was that this was the expected release style for every company at the time.

The quarterly release cadence was a novel idea that revolutionized how companies would develop and test their code. The code changes were smaller in nature and the teams evolved their thinking from a new product every year to a new subversion every quarter. Some user experience changes may be introduced, but most of the user experience in the application would remain the same from release to release. Another major benefit to the increased release frequency was the significant reduction in lead time, which is the time it takes to go from a feature being requested to being available in production.

Alongside the release parties were two very important processes when issues would arise during the release:

• Root-cause analysis (RCA)
• Post-mortem

An RCA would occur anytime there was a significant issue in production that would halt or severely affect the functionality or availability of the application. Often, the RCA process would start with the teams analyzing what was wrong, fixing the issue, validating that the fix worked, and then documenting how the issue arose and what the root cause was. Every release party would result in at least one RCA taking place, and would exponentially increase in number relative to the total amount of production servers involved in the release party.

The post-mortem was a retrospective process after the release was completed and the teams were confident in production operating as expected. The release captain would gather any and all information related to RCAs, bugs, errors, and so on, and create the required documentation and tickets. At the end of the post-mortem, a weekend support team would be briefed on the release party outcome and any items needing to be monitored.

The desire to automate the release of the application had been a central focus of every engineering organizations for years Automation was seen as the best way to enforce reliability and repeatability into the release process, and most of the common tools in use today were created with the intention and purpose of release automation. These tools, and really the underlying processes they address, intend to solve two major concepts in the software development life cycle: deliveries and deployments.