Building workflows that are repeatable and never lost
As DevOps engineers, we write tooling all the time. In small shops, many times, these are sets of scripts. In large shops, these are complicated systems.
As you may have gleaned from the introduction, I believe that tool execution should always occur in a centralized service, regardless of scale. A basic service is easy to write, and you can expand and replace it as new needs arise.
But to make a workflow service work, two key concepts must be true of the workflows you create, as follows:
- They must be repeatable.
- They cannot be lost.
The first concept is that running a workflow more than once on the same infrastructure should produce the same result. We called this idempotency, borrowing the computer science term.
The second is that a workflow cannot be lost. If a tool creates a workflow to be executed by a system and the tool dies, the tool must be able to know that the workflow is running and resume...