Introduction
In the last decade, several comprehensive paradigm shifts have taken place in the software industry. These changes have made it possible for millions of people to chat on their mobile phones at the same time and stream their favorite movies while traveling throughout the world. When changes in software development and operations are reviewed, two prominent, mutually reinforcing paradigms come to the forefront: DevOps and cloud-native architecture. DevOps created a cultural change by establishing more open communication between teams. This cultural change led to practices such as continuous integration, testing, and deployment, which shaped today's software development methodology. Likewise, cloud-native architecture created an open environment with scalable microservices capable of serving millions of customers. In order to manage this scalability, container technologies have evolved for the development, testing, and deployment of applications. These two paradigm shifts have enabled today's robust, scalable, and easy-to-manage software applications to change the technological, social, and industrial face of the world.
Before diving into innovative software development methods, let's have a glance at a conventional approach. Traditionally, software development was similar to manufacturing a passenger aircraft. Considerable investments in infrastructure and personnel followed the collection of requirements, design, and planning. There were teams identical to production-line engineers and workers that specialized in a particular topic and delivered parts of the aircraft for the next stages. There were even companies with organizational structures including teams named "production line." Prodigious output was delivered to customers following formal acceptance tests. After that, it was the customer's responsibility to make the aircraft fly with its team of engineers and operators. Further requests and upgrades were part of another substantial project.
Today, software development has evolved and become more customer-oriented. It has moved away from making customers buy software products by having customers subscribe to software services. A similar thing could be said for the aircraft analogy: software applications are smaller and more flexible, like drones. Requirements are collected at every stage, and the product is configured to the customer's environment "on the fly." Since customers also buy the product as a service, maintenance and keeping drones in the air is the job of the producer. In order to manage these services, microservice orchestrators such as Kubernetes make these small drones run in harmony to achieve more complex air shows. These changes in software development and operations owe their success to the cultural shift of DevOps and containerized cloud-native technologies.
DevOps' Effect on Industry
Leading companies such as PayPal, Facebook, and Netflix have very strong DevOps success stories that have evolved over the years. For instance, Paypal has more than 200 million active users, with nearly 5,000 developers. In 2013, creating a new application on PayPal required opening dozens of tickets and following their complex statuses for months instead of writing code. To resolve this problem, PayPal developed a software development lifecycle system to manage the complete lifecycle of software, going from planning to production in a couple of weeks.
Likewise, Facebook focused on code ownership, automation, and continuous improvement way before DevOps became popular. Today, Facebook uses the Chef configuration management tool to manage all of its infrastructure and backend systems. Similarly, Netflix created an environment where thousands of changes are made to production each day. It both decreased the time taken to fix problems and increased its market responsiveness.
When old and new software development practices are compared, it is evident that the old mindset of conventional software development is doomed to fail. Running scalable, reliable, and robust applications on cloud providers that can scale to serve millions of customers requires learning and applying new methodologies. The basics of these methodologies include learning the basics of DevOps culture and toolchain and container technology. Following that, it is essential to learn and exercise how to install, configure, scale, and monitor containerized applications inside the de facto container orchestrator, Kubernetes.
In this chapter, the inception of the DevOps cultural shift and its value toolchain are explored. How DevOps changed the software development environment and potential benefits for organizations are covered. Following that, every step of a complete DevOps toolchain will be discussed, starting from the plan for a software project to monitoring an installed application. All toolchain steps are presented and experimented on with a modern cloud-native application suitable for today's software trends.
DevOps Culture and its Benefits
Traditional software development focused on planning, developing, testing, and delivering software systems with separate teams focused on the respective areas. The outputs and expectations of teams were defined in detail beforehand, and each team was responsibile for its deliverables. For instance, the planning team, consisting of seasoned planning managers and industrial engineers, would calculate the working-hour requirements and delivery dates for their output. The development team would create software, and the testing team would test the output of the development team.
Finally, the delivery team would visit a customer on site and install the software systems according to the customer's needs. These consecutive stages were huge, and they were done with minimal inter-team communication. The state of mind was based on not interfering with the business of other teams involved in the development process. With this style of development, many IT projects were undertaken; however, most of them failed. To make it clear with actual numbers, according to research on "The Impact of Business Requirements on the Success of Technology Projects", by IAG Consulting, in 2008, 68% of IT projects failed as they were impractical due to lack of communication between the various teams.
What made the software projects impractical was a lack of proper requirement analysis and inter-team communication. Planning and consulting teams would collect requirements without the collaboration of development teams. With the same kind of flawed approach, development teams did not cooperate with the operations teams responsible for configuring, installing, and monitoring applications for customers. This lack of formal inter-team communication resulted in development teams having minimal knowledge about the runtime environment. On the other hand, operations teams had practically no concrete understanding of the requirements and features of the applications they were deploying. With enormous barriers between these teams, they created applications that did not concurrently consider the runtime environment and software requirements. Consequently, both development and operations teams were held responsible for many failures, thus leading to financial losses.
As the term DevOps derives from the combination of development and operations, the DevOps culture came in to being to increase the collaboration between development and operations teams. With DevOps' cultural change, companies now form DevOps teams consisting of engineers from development and organizational backgrounds. These new teams help developers to realize both operational and customer requirements. On the other hand, operations engineers gain insights into applications and development requirements. With the barriers between teams having collapsed, requirements are collected efficiently, quality is fostered, and lead times are reduced. The benefits of this cultural shift have led to DevOps being adopted in organizations of various sizes, from start-ups to enterprise companies.
Note
The term DevOps was coined by Patrick Debois in 2009 and was first used at a devopsdays conference in Belgium. Devopsdays focuses on software development and IT infrastructure operations and events are organized worldwide throughout the year. You can read more about this at:
With the successful implementation of DevOps, not only has communication between teams increased, but software delivery speed, reliability, and scalability have also been enhanced. Firstly, DevOps culture indicates better requirement collection and better utilization of those requirements in the product design. Therefore, it is expected to have decreased the time taken to deliver new product features to the market. Secondly, with continuous integration and testing, more robust and reliable applications are expected. Finally, DevOps also enhances operations including configuring, deploying, and monitoring.
With infrastructure-as-a-code practices and metrics from production environments, it is expected to have scalable applications. DevOps culture is capable of providing organizations with various advantages. However, before implementation, understanding the current company's culture and creating a feasible action plan for introducing DevOps is crucial. In the following sections, the DevOps toolchain is explained in detail to illustrate how DevOps' cultural shift has evolved into a value chain for software development.