Download code from GitHub
DevOps can be seen as an extension of the successful and well established Agile methodology. Bringing operations into the picture helps the advance from continuous (Agile) development to integration to deployment, but more importantly it helps build a better working environment, one with stronger cross-team relationships.
If I had to describe DevOps in one word, it would be collaboration. The genuine willingness of both the Dev and Ops camps to work together is the foundation, the most important aspect of the philosophy.
DevOps appears as the meeting point in the following diagram:
In this chapter we will go through the following topics:
What is DevOps?
Questions you should ask yourself before adopting it
So, let us examine the various principle characteristics of a DevOps environment.
What follows is a series of generally accepted definitions, invariably mixed with personal opinions - you have been warned.
The alignment of effort toward increasing system performance and stability, reducing the time it takes to deploy or improving the overall quality of the product, will result in happier customers and proud engineers.
The goal needs to be repeated, clarified, and simplified until it is fully understood, argued against, and eventually committed to by everybody.
DevOps shifts focus away from self-interest and toward that goal. It directs praise at group achievements rather than those of the individual; KPIs and Employee of the Month initiatives perhaps not so much.
Allow people to look at the bigger picture past the realm of their cubicle. Trust them.
The chances are you have already heard stories or read books about the notorious organizational silos.
In the worst case, it would be somebody who refuses to let go and often becomes the main bottleneck in a development life cycle. They can be fiercely territorial, safe-guarding what exclusive knowledge they might have in a given field, likely (I speculate) because this provides them with a sense of importance, further catering to their ego.
On the other hand, there are also examples of people who find themselves in a silo purely due to unfortunate circumstances. My respect goes out to the many engineers stuck with supporting inherited legacy systems all by themselves.
Fortunately, DevOps blurs such borders of expertise with concepts like cross-functional teams and full-stack engineers. It is important to note here that this does not translate into an opportunity to cut costs by expecting people to be tech ninja experts at every single thing (which in real life equates to preceding average). But, as in one of those Venn diagrams, it is the cross-over between a Dev and an Ops set of skills.
Silos are avoided by encouraging knowledge sharing. Peer reviews, demo stand-ups, or shared documentation are a few ways to ensure that no task or piece of know-how is limited to a specific person. You could even adopt Pair Programming. It seems a bit heavy, but it evidently works!
Should developers be given production access?
There are good reasons for maintaining strict role-based permissions; one of them is security another is integrity. This standpoint remains valid for as long as we maintain the stereotype of the developer who is so used to working in devlocal; to them, concepts such as passphrase-protected SSH keys or not manually editing all of the files take a back seat.
In the era of DevOps, this is no longer the case. Shared knowledge and responsibility means operations engineers can rely on their developer colleagues to follow the same code of conduct when working in critical, production environments.
Dev and Ops teams have access to the same set of tools and environments. Deployments are no longer a special task reserved for the Ops team and scheduled days in advance.
In a team with such knowledge-sharing habits, I, as an operations engineer, can be confident about my Dev colleague's ability to perform my tasks, and vice versa.
Developers participate in the on-call rota, supporting the software they produce.
This is not to be seen as an additional burden, but as a sign of trust and an opportunity to increase collaboration. Nobody is throwing code over the wall anymore. Responsibility and a sense of autonomy motivates people to do more than is expected of them.
As we spend more time talking to each other about the challenges we face and the problems we are trying to solve, our mutual respect grows.
This manifests itself in developers seeking input from the Operations team from the early stages of the software development process or in Ops tools being built to meet developers' needs.
Ops who think like Devs. Devs who think like Ops | ||
| --John Allspaw and Paul Hammond, Velocity | ||
A DevOps environment is built on such respect. It is a place where every opinion matters, where people can and do openly question decisions in the interest of the best solution to a problem. This is a powerful indicator of one's commitment toward the common goal I mentioned earlier.
To draw an overly simplified conclusion from A. Maslow's Theory of Motivation, you are less likely to think about poetry when hungry. In other words, a team with basic needs will be solving basic problems.
Automating routine and mundane tasks allows engineers to concentrate on the more complex, higher-value ones. Also, people get bored, cut corners, and make mistakes – computers tend not to do so.
Describing infrastructure as code has the following advantages:
It can be kept under version control
It is easily shared with others to re-use or reproduce
It serves as a very useful diary of what you did and how exactly you did it
Provisioning cloud environments becomes trivial (for example, with Terraform or CloudFormation)
It makes modern Configuration Management possible
At any rate, I suspect anybody managing more than 10 servers is already codifying infrastructure in some way or another.
Measure All The Things! | ||
| --Actual DevOps slogan | ||
Storage is cheap. Develop the habit of gathering copious amounts of measurements and making those easily accessible across your organization. The more visibility engineers have into the performance of their infrastructure and applications, the more adequate their decisions will be in critical situations.
Graphs can convey a great deal of information, look rather cool on big screens, and the human mind has been proven excellent at recognizing patterns.
Another important role of metrics data is in performance optimization.
The trickiest part of speeding up a program is not doing it, but deciding whether it's worth doing at all...Part of the problem is that optimization is hard to do well. It's frighteningly easy to devolve into superstitious ritual and rationalization. | ||
| --Mature Optimization, Carlos Bueno | ||
To avoid falling prey to confirmation bias, you need an objective method of assessing your systems before and after attempting any optimization. Use those metrics; it is hard to argue with (valid) data.
On the subject of validity, please do calibrate your instruments regularly, sanity-check output and make sure what you think you are showing is what your colleagues think they are seeing (ref: https://mars.jpl.nasa.gov/msp98/news/mco990930.html).
The Observe, Orient, Decide, and Act (OODA) loop is a concept developed by Col. J. Boyd that shows the value in one's ability to adapt to ever-changing circumstances.
Faced with unforgiving (and productive) competition, organizations should be able to rapidly react to dynamic market conditions.
This is probably best illustrated with the old Kodak and Netflix tales. The former after having been wildly successful is said to have failed to adapt to the new trends in their sector, causing the brand to gradually fade away. In contrast, Netflix keeps on skillfully molding their product to match the new ways in which we consume digital content. They completely transformed their infrastructure, shared some clever, new and somewhat controversial practices plus a ton of great software. Be like Netflix.
Continuous Integration and Delivery is essentially OODA in practice. Teams continuously integrate relatively small code changes, delivering releases more often, thus getting feedback from their users much quicker. The type of feedback needed by an organization to be able to adequately respond to an ever changing market.
None of the preceding suggests however that one should aim to become a release hero, rushing things into Production, setting it on fire twice a week. A CI/CDframework still implies the usual strict code review and test processes, despite how often you deploy. Though code reviews and testing require much less time and effort as typically the more frequent the deployments, the smaller the code changes.
Naturally, more experimentation is likely to increase the probability of error.
I doubt this fact comes as a surprise to anybody; what might surprise you, however, is the advice to accept an additional, positive angle to failure.
Recall the video nerds from the previous section. Well, they didn't exactly breeze through all that change without casualties. I hereby spare you the Edison quotes; however, trial and error is indeed a valid form of the scientific method, and the DevOps processes serve as a great enabler to those who would agree.
In other words, an organization should encourage people to keep on challenging and improving the current state of affairs while also allowing them to openly talk about the times when things went wrong.
But dealing with experimentation failures is possibly the more romantic side of the story compared to the cold, harsh reality of day-to-day operations.
Systems fail. I would like to think most of us have come to accept that fact along with the chain of thought it provokes:
we do not always know as much as we think we do:
"Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case...
After an accident, practitioner actions may be regarded as errors or violations, but these evaluations are heavily biased by hindsight and ignore the other driving forces..."
--How Complex Systems Fail, R.I. Cook Excelsior! Or how, in our long-standing pursuit of social dominance, we seem to have developed the convenient belief that following an event we not only know exactly what and how it happened but also why. This peculiar phenomenon has already been explained rather well by D. Kahneman in Thinking Fast and Slow; I will just add that indeed one often hears of overconfident characters who point fingers at their colleagues based on what appears to them as a coherent storyline.
The truth of the matter is this: we were not there. And keeping the details we now know and those known at the time separated is not an easy task.
Blaming is of zero value:
Etsy and the likes in our community have shared enough observations to suggest that negative reinforcement as a strategy for reducing human error is less than optimal.
With the adoption of DevOps, we accept that people generally come to work every day with the intention to perform to the best of their abilities and in the interest of the organization. After an outage, we begin our analysis with the assumption that the operator has acted in the best possible way given the circumstances and information available to them at the time. We focus on what could have led to them making the given decisions, their thought process, the flow of events, and whether any of these can be improved.
Resilience can be accumulated:
"What does not kill us..."mithridatism or Nassim Taleb's concept of antifragility are all expressive of the idea that we get better at dealing with negative experiences as we encounter them, and what's more, we should look for them every now and again.
We can train ourselves and our systems to recover from errors faster or even better to continue operating despite them. One way to achieve this is with controlled (and with practice, less controlled) outages.
With the right monitoring and auditing tools in place, every abnormal activity offers us a more intimate view of our applications and infrastructure.
Now that I have bestowed upon you, my dear reader, the secret to a better life through DevOps, let us concern ourselves with the latter part of the title of this chapter.
I fail to see a reason why one should not. Some seven or so years have passed since the inception of the idea of DevOps, and the amount of evidence of its effectiveness has been growing steadily. Having the respected Agile framework at its base further adds to its credibility and perhaps helps explain a good part of its success.
That is not to say there are not considerations to be taken into account however. The critical thinker within you, would want to ask a question or two prior to embarking on such a cultural coup d'état.
Did you just finish adopting Lean or Agile Development? What else has been going on in the team? Is now the best time for yet another cry for change?
Altering our habits makes us uneasy; it takes some time to adjust. Your perseverance is laudable, and pursuing DevOps as the next level of team collaboration is often the right choice.
There is no need to give it up altogether; perhaps put it on hold for a moment.
Look around you. Those faces, those different personalities, can you picture them all together singing Kumbaya? Maybe yes, maybe no, or not yet.
Please do not e-mail an anonymous staff survey. Get everybody in a room, lay your DevOps propaganda out, and gauge their reactions.
You will need everyone to fully understand the concepts, acknowledge the challenges, and accept the sacrifices for this to work. There can neither be exceptions nor ambiguity.
All of this requires a great degree of cultural change, which a team should be prepared for.
What would it take to change the current mentality? How much of a disturbance you would need to cause? What degree of backlash do you expect?
While I am not suggesting this as an excuse to put up with the status quo, I beg you maintain a pragmatic view of the situation.
Your type of organization might be better suited for a process of evolution rather than a revolution.
How would you score your current processes? Would you say your cross-team communication is satisfactory? You regularly meet business expectations? You have already automated most of your workflow?
It sounds like you are doing fine as it is; you might already have some DevOps in your team without realizing it. The point is that it could be a better use of resources if you were to concentrate on optimizing elsewhere, solving other, more pressing problems at this time.
Now that you have been through a yet another interpretation of the ideas behind DevOps, if you feel those match your way of thinking and the final few questions did not raise any concerns, then we can safely transition to the more technical topics where we put principles into practice.
First, we explored the main ideas contained in the DevOps philosophy, followed by a few questions aimed at helping you construct a more objective perspective when it comes to adopting DevOps within your organization.
We have seen that DevOps is an effective combination of some older, proven Agile concepts and other more recently developed ones, and that it teaches us how to build better teams who write better software, get results faster, and collaborate effortlessly in an environment that encourages experimentation without compromising stability.
Now that we have covered the theory, the next chapter takes us into the practical application of DevOps. We are going to begin with examples of deploying infrastructure as code in the cloud.
