What is a delivery process?
The time between the release parties seemed to get shorter with each quarter, especially when the requirements continued to grow. The different engineering groups were becoming smaller as engineers were burning out and leaving. The system administrator team needed to quickly figure out what had to be fixed and what could be automated. But after every release party, the backlog of work items continued to pile up, resulting in a mountain of technical debt.
The recent move from semi-annual releases to quarterly releases required a massive operations team overhaul. Some of the more senior infrastructure and engineering team members moved to a new group focused on automating the release process. But before they could automate anything, they needed to understand what a release actually consisted of. Since no team or person knew all of the different release requirements, the next quarter was spent on research and documentation.
What they found was that every step related to the lifecycle of the artifact, from development to production release, needed to be automated. This lifecycle, which included all pre-deployment, deployment, and post-deployment requirements, was considered a delivery. What they were surprised by was finding out that one execution of the delivery process took 3 months to complete. To make matters worse, every step in the delivery process had never been documented or defined. Every member of every team that was involved in the delivery process had a different reason for why each of their steps were required.
After the team had completed their initial documentation process and also after experiencing the recent release party, they were ready to start automating. But what they were unsure of was whether or not they should build out the process or if they should look for a tool to do it for them.
At first, they researched the available tools in the market that might give them a foundation to build the new process on top of. One issue they found was that the more common tools were related specifically to either Windows or Linux, but not both. The other tools that they had found were scalable across the different systems, but they required significant ownership and hosting requirenments. Considering the short timeline and technology requirements, any tool that supported multiple systems and could be highly customized and extended through scripting would be best.
The system administrator team decided that it would be best to split up and tackle different requirements. Some of the team focused their attention on running tooling proof of concepts. The rest of the team would focus on building scripts to support the rest of the engineering organization. The initial iteration of the new deployment process would be focused mainly on the ability to build and execute the automation scripts. Once that was built out, the next iteration improvement would focus on turning the scripts into templates for scalability.
The first piece to automate was the deployment of the release candidate artifact to the test server. This required bootstrapping the server (resetting it to an optimal state, adding the desired environment variables, adding any required software or patches, and so on). Then they would pull down the artifact from a designated network share, expand it, and upgrade the application on the server. After that was completed, the automation process could then email the QA team for them to start their testing requirements.
With the development deployment completed, automation process would be directed at a staging environment. This environment contained multiple production-like servers , allowing the operations team to practice the deployment process. Automating this requirement, meant that the deployment scripts had to affect the network configuration, as well as the application server. After the server was bootstrapped and reset, the testing process could then be run to validate the application health. But, to make the automation more scalable, the team would need to have the scripts run remotely, pushing down that artifact instead. The remote execution behavior would allow for a larger deployment set to be run in parallel.
The last part that the team wanted to automate was the post-deployment verification step. This step could be run remotely and would pass data to the application. This would allow for both a network connectivity check and a desired functionality check.
The team would need to test out the automation process in production
One of the biggest issues that any engineering organization must deal with is technical debt. Technical debt is the cost of any rework that is caused by pursuing easy or limited solutions. And what makes technical debt grow is when engineering organizations work as a set of disparate units. This causes compound interest, since no central team will be able to maintain the cross-team technical debt.
ventually, the creditor comes to collect the technical debt and, depending on the communication styles of the teams and how long the debt has been avoided, the hole is almost too deep to climb out of. Technical will often go unnoticed for a few years few years until the creditor comes to collect. With regard to a engineering organization's technical debt, the creditor is often business or market requirements. Most recently, smaller and more agile start-ups disrupt the market, taking market share, and causing panic for the bigger players.
With the potential of technical debt resulting in a form of technical bankruptcy, many companies make radical decisions. Sometimes they decide to create new teams focused on a new company direction. Other times they will replace the management team for a "fresh perspective". But in any case, the goal is to repay the technical debt as quickly as possible.
A common place to find technical debt is in engineering supporting processes. For the system administration team in the analogy, most of the technical debt was associated to their release practice. Although they had a relatively automated deployment, they found that most of the manual steps were things that occurred before and after. As as result, the team realized that the biggest source of technical debt was their delivery process.
Any desire to automate a process must first start with a requirements gathering process. With the requirements gathered, a team can then pursue a minimally viable product. Part of the requirements gathering process is being able to define what the immediate needs were and which capabilities could be added at a later time. A minimally viable product is exactly as it sounds, a product that meets minimal requirements to be viable. In the analogy, the items that would be required in the MVP were server bootstrapping, artifact deployment, and network management. These functionalities would have the highest level of impact on the technical debt and also on the main problem areas throughout the current delivery and deployment process. Features such as running and evaluating tests, approval steps, dynamic environment creation, and traffic manipulation would be brought in over time.
Building, testing, and iterating are the common development cycles that any engineering team will need to go through. But the moment that any process is automated, the team responsible for the automation will need to consider scalability. Once another another automation use case is discovered, the automation tooling must be scaled to accomodate. The term often associated with scaling across use cases is known as onboarding. And a requirement to onboard other use cases or teams immediately develops a need for a central management team. That team will have the goal of supporting, improving, troubleshooting, and onboarding the solution for the foreseable future. Eventually, the automation process becomes a core support tool that must be reliable, scalable, and offer repeatable outcomes.