Defining Agile's core implementation concepts
You have probably already surmised that the lightweight methodologies introduced in previous sections have more in common than not. This section consolidates and elaborates on the common principles and practices that came from the lightweight software development methodologies.
I have ordered the concepts alphabetically, to avoid an inaccurate view that one concept has more or less importance than any of the others. All of these principles and practices have found wide-scale acceptance. The order of implementation within an organization is situational and usually best implemented at the team level – for reasons I'll explain in Part Two of this book.
Backlogs
In the traditional model business analysts document business and user requirements in some form of business or requirements specification, while members of the development team document functional and nonfunctional requirements and develop related technical specifications. All of that goes away on agile-based software development projects. Instead, a product owner maintains a backlog of identified requirements, typically documented in the form of epics, stories, and features. While the development team members help the product owner refine the product backlog, only the product owner has the authority to set priorities within the backlog.
Co-location
Modern electronic communication, collaboration, and source control management tools allow software development teams to work in virtual environments and across multiple locations. I've managed more than my fair share of virtual software development projects, both large and small, and I've been able to make them work.
However, an ideal situation is to locate all the development team members in a single location, working together in one room large enough to accommodate the entire team. If the organization has the resources, a common practice is to place a central table with sufficient size to sit all the team members around the table with their computers, so that they can face each other and talk comfortably while they're working. Also, the room's walls should all have whiteboards on them so that team members can conduct multiple collaborative working sessions at the boards simultaneously. The room should be dedicated to the team so they can keep the information on their whiteboards and charts out in the open at all times.
The team members also need time and space to work independently. So, again, if the organization has the resources, the ideal situation is to have individual work cubes in or adjacent to the room. The main point is that the teams can come together to work through technical implementation concerns or address problems that crop up during the project.
In Part Two of this book, you'll discover that having multiple Scrum teams associated with the development of a single product is not uncommon. Each team works as an independent group, and they may work at other facilities. However, each separate team should work together as a single unit and co-located in a single facility with a dedicated room to work and collaborate as a team.
CI/CD pipelines
The acronym CI/CD or CICD refers to the combination of continuous integration and either continuous delivery or continuous deployment practices. Continuous integration preceded the concepts of continuous delivery and continuous deployment. The term Continuous Integration was originally described by Kent Beck as part of his 12 Extreme Programming development processes. Continuous delivery is the automation of provisioning development and test environments on-demand and with proper configurations. Continuous deployment takes CI one step further to include automated provisioning and configuration in a production deployment environment. The next two subsections provide more details about these two specific areas of automation, CI and CD.
Continuous Integration
Bugs in software code are pretty much a given. The longer a development team waits to integrate and test their source, the more bugs accumulate. Also, the accumulation of both code and bugs makes the task of debugging more complex, difficult, and time-consuming. More frequent integration and testing make debugging less difficult. The causes of bugs are more easily found and more easily fixed.
Continuous Integration is the practice of frequently integrating code into a shared repository – at a minimum, several times a day. More frequent check-ins are better than less frequent check-ins. For larger sections of code, a good practice is to use stubs and API placeholders (a.k.a. skeletons) so that your code can be tested and checked into the source code repository more frequently. This allows other programmers on the team to see what you are doing and even assist with the development of the functionality delivered to the stubs and APIs.
The development team should always work from a single source code repository. Automating the build process is an important component of CI. The automation process includes compiling source code files, packaging compiled files, producing application installers, and defining the database schemas. The more repeatable, self-testing, and automated the build process, the less likely it will be that the process will introduce bugs.
Ideally, developers initiate the build process every time they check their code into the SCM system, and the build process should take less than 10 minutes, including the execution of automated tests. If the build process is manually intensive and time-consuming, developers won't do it frequently, leading to the previously mentioned bug accumulation issues.
Another good practice is to have every commit build on a dedicated integration server. Cloud-based platforms and virtualized development environments make this strategy increasingly practical. The dedicated integration system offers better visibility to available code and provides a staging environment for subsequent testing.
Continuous delivery/deployment
There are several important elements of the concepts and practices behind the implementation of continuous delivery and deployment capabilities. As a component of agility, iterative development practices generate frequent releases of potentially shippable products that can be made available to customers in a predictable cycle. The frequent availability of useful functional software impacts the organization's delivery capabilities.
Specifically, frequent releases create potential issues in the areas of provisioning. Each new release of software has the potential to impact the organization's infrastructure with new demands for computing equipment, networks, database management systems, backup and storage capabilities, security applications, and other software applications.
Continuous Delivery is the term used to define an IT organization's ability to implement changes on demand. Such changes include the introduction of new features, bug fixes, software, and system configuration changes. Provisioning concerns span the development, testing, and production environments, encompassing all software and related infrastructure requirements.
The goal of Continuous Delivery is to implement changes to the development, test, and production environments safely, rapidly, and sustainably. As an example, continuous delivery capabilities make it practical to release software at extremely short intervals, even multiple times a day.
Cloud-based Platform as a Service (PaaS) offerings make continuous delivery both economical and practical for many organizations. PaaS providers offer their customers a virtualized application development and hosting platform as a cost-effective alternative to building and maintaining dedicated, in-house IT infrastructures. But the same concepts and capabilities are possible within an organization's data center with the correct technologies and practices.
Continuous delivery concepts extend the automated build and integration test capabilities out into the test environments. They also generally include the ability to provision and configure necessary infrastructure resources on demand. Conceptually, continuous delivery supports the development and testing environments. After the completion of testing, the development team manually pushes the released software into the production environments. The customer's IT operations team then provisions resources and configures the hardware and software for the new release.
Continuous deployment takes the automated provisioning and configuration capabilities of continuous delivery out into the production environments. Humans are not involved, and only a failure in the automated process stops the delivery of the release. With the implementation of both continuous delivery and deployment capabilities, developers have the potential to release new product functionality within minutes of pushing their code into the test environments.
Continuous deployment capabilities create a new set of potential problems by releasing software updates to customers who don't value, or understand, or are not yet willing to adopt the changes. Also, the functionality may not apply to every type of user. Therefore, it's not unusual to set up rules that limit the deployment of automated production releases to stages or to specific communities of users.
Cross-functional
The implementation of iterative development releases by small teams is only possible if the teams have the resources and skills required to develop and test the product's features. As the teams are stood up, they must collectively have the skills to work on the product. Over time, individuals should strive to cross-train on the skills they don't have but that are necessary for the development of the product. This concept is called "multi learning" and is key to the team's self-sufficiency.
Customer-centric
The third value in the Agile Manifesto says customer collaboration is more important than contract negotiations [Kent Beck, et al. © 2001. Manifesto for Agile Software Development. http://agilemanifesto.org/. Accessed 10 November 2019]. A contract scenario creates situations where the customer defines high-level requirements and then, except for brief requirement gathering activities, disappears for the remaining duration of the project. Moreover, contract negotiations set up adversarial relationships, and not the collaborative relationships that support the iterative and evolutionary development aspects of agile.
Incremental and iterative development (IID) practices help simplify development by reducing the size and complexity of code, making it easier to find and debug any errors in the application. However, the IID practices also allow several very important customer alignment and collaboration opportunities.
First, the customers and end-users can look at the product frequently to ensure the implementation of features that meet their needs. Invariably, customers get new insights into how a software product can better support their needs only after they have a chance to work with a prototype or new release. The frequent releases of agile provide more opportunities for customers to get these insights.
Second, if the customer judges the software features as insufficient in some manner, the customer's priorities have changed, or their insights after viewing the software lead to new requirements, the development team can quickly adjust to meet the new requirements. In the traditional model, change requests were not desirable as they had a direct impact on the authorized constraints of the project. In other words, changes in scope cause negative impacts on project budgets, resource allocations, and schedules.
Iterative and Incremental Development (IID)
The phrase iterative and incremental development describes the agile approach practitioners take to deliver frequent releases of potentially shippable products. I have found there is often confusion about what the terms iterative and incremental mean.
The term iterative means development is broken up into limited time-boxed durations, typically of 4 weeks or less. In a modern DevOps environment with continuous delivery and deployment capabilities, iterations are measured in hours or minutes. Iterative development reduces complexity and work in progress, which are important elements in implementing lean software development concepts.
The term incremental refers to the concept that the product evolves incrementally with each development iteration. In other words, the term increment implies the release of a new slice of functionality with each iteration.
Iterative and incremental go together because the sole purpose of development, regardless of the methodology employed, is to create something that customers ultimately value. In that context, new increments of product functionality are the deliverables of the iterative development process.
Pair programming
There are several important elements associated with agile practices that allow the development team to operate rapidly, efficiently, and productively. Some of the most important elements include transparency, safety, perspective, and inspection.
An effective strategy to address all four of these elements is pair programming. In pair programming, two developers sit side by side and work together. One programmer has the role of driver, while the other takes on the role of an observer, sometimes called the navigator.
As the driver develops the code, they are heads-down and operating at a tactical/ implementation level. The observer inspects the developer's code in real-time to help resolve any bugs and consider more efficient development approaches. The observer also evaluates the code in terms of addressing the strategic goals of the feature within the application. The developers change roles frequently to stay fresh and engaged.
Potentially shippable products
The first principle stated in the Agile Manifesto is Our highest priority is to satisfy the customer through early and continuous delivery of valuable software [Kent Beck, et al. © 2001. Manifesto for Agile Software Development. http://agilemanifesto.org/. Accessed 10 November 2019.] Agile practitioners employ IID practices in support of this principle.
However, it does not matter how frequently the development team completes new updates if the output has no functional value to the customer. Therefore, a key component of most agile practices is the concept of providing a potentially shippable product with every development iteration.
That does not mean to imply the product owner must release the product of an iteration to customers and end-users. There may be practical reasons not to do so until full business process functionality is available, or the customer is in a position to accept the product into production.
Prototyping
I noted previously that, often, customers don't know what they want until they see the product and have a chance to work with it. This is the point of having frequent customer reviews, as often as every iterative release. At the beginning of a project, there may not be much to show. Moreover, the team may want to show potential product concepts before devoting much time to developing the features.
Prototyping is an approach to quickly and cost-effectively build a mock-up of a particular product or feature. The mock-up could be as simple as a screen or user face design, or more complex with data fields, data, and algorithms that support the proposed application functionality. The prototypes are shown to customers before the implementation of the feature into a production release of the product.
An important concept of prototyping is that all prototypes are, by definition, throwaway activities. In other words, there should never be an expectation that a prototype must be deployed to a customer. Such an expectation gets in the way of the concept that failures are not bad, or to be avoided, as they provide opportunities for learning. If nothing else, the developers now know what not to do in the future.
Retrospectives
The twelfth and final principle of the Agile Manifesto states: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly [Kent Beck, et al. © 2001. Manifesto for Agile Software Development. http://agilemanifesto.org/. Accessed 10 November 2019]. Retrospectives support this principle.
During a retrospective, the development team looks back at the past iteration to assess what worked well, what didn't work out so well, and what they could do differently in the future, starting with the very next iteration. Retrospectives only work and have value when the development team members feel free to speak out. In other words, they must work in an environment where team members are respectful, open, and honest, with complete transparency about all customer and product information.
Safety
Pair programming, prototyping, and retrospectives all create situations for potential tension and conflict. The development team member must be open to alternative views, know that failure is not going to get them in trouble, and that they should feel free to speak out during retrospectives and other team collaboration activities. That requires the development of an environment that stresses respect and safety for its team members.
The concept of safety is 180° out from the command and control environments dictated by the traditional development methodologies. The plan-driven and linear-sequential approach to development in the traditional model created cultures where team members followed edicts driven from above. In contrast, agile encourages a bottom-up approach to planning and executing work that puts the development team and the product owner in the driver's seat for decision-making.
Self-organizing
Having team members with multiple useful skills gives the team more flexibility in the assignment of work. Development teams can expect that each iteration of work has different task requirements. If the team employs specialists and does not cross-train – both to develop missing skills and provide more flexibility in individual work assignments, they have limited ability to assign work on demand. Instead, they find themselves having to wait to take on work until the required specialist is available.
From the context of the product backlog, having teams filled with specialists causes delays in the assignment of work and bottlenecks that, in turn, creates too much work in progress. Accumulated work in progress is a form of waste and is the antithesis of lean development organizations.
Small teams
Effective software development teams must have the resources they need to deliver incremental slices of new product functionality every iteration. If a team is too small, they likely won't have the skills and resources to deliver. If the team is too large, the coordination of work grows significantly larger and more complex.
Ken Schwaber and Jeff Sutherland, in the Scrum Guide, state teams should not be smaller than three members and not greater than nine. Jeff Bezos, the CEO of Amazon.com, is famous for noting that no matter the size of the company, individual teams should never get large enough to consume more than two pizzas in a sitting. (for example, five to seven people) [Deutschman, A. (2004) Inside the Mind of Jeff Bezos. Pizza Teams and Terabytes. https://www.fastcompany.com/50106/inside-mind-jeff-bezos-5. Accessed November 8, 2019]. UK anthropology professor R.I.M. Dunbar, in his paper titled Co-Evolution of Neocortex Size, Group Size, and Language in Humans, cites the maximum useful team size is limited to no more than 10 to 12 people [Dunbar, R. I. M. (1993). Coevolution of neocortical size, group size and language in humans. Behavioral and Brain Sciences 16 (4): 681-735], [Buys, C.J. & Larsen, K.L. (1979). Human sympathy groups. Psychological Report. 45: 547-553]. The Scaled Agile Framework (SAFe™), another scaled Scrum approach discussed later in this book, recommends that development teams stay in the range of 5 to 11 people. [Corporate (2019) Scaled Agile. Agile Teams. https://www.scaledagileframework.com/agile-teams/. Accessed November 8, 2019].
Source Code Management (SCM)
Developers work on code as individuals and in groups as part of a larger team. The developers implement changes to their code incrementally, not sure if the individual changes can deliver an intended result. In such cases, they may need to retrieve the original code to start over. That is the primary purpose of an SCM tool, to serve as a repository for all source code and related artifacts, and to automate version control over each of the stored artifacts so that developers can roll back to whatever versions they require.
The SCM tool also makes it possible for developers to find and check out the latest version of the code under development. SCM is an important capability as the developers must have confidence that they are working from the most current code set. If they pick up an old code set, there are likely to be numerous changes that can negatively impact the code they are developing. SCM tools also make it easy for the members of the development team to see the completed work to date, work in the queue, and work in progress.
Stories
There are many approaches to documenting business and user requirements. Under traditional development practices, requirements documentation tends to have a focus on analyzing what a product must do to fulfill business and user needs. In other words, the focus of requirements analysis is on the product. In contrast, story-based formats provide a brief elaboration on what customers and end-users want to do. In other words, what are the capabilities they need to do the work of their organizations?
It's a subtle distinction but think of it this way: customers don't care as much about the list of product features as they care about what they can do with the product's features.
The basic format of a story is as follows:
As a <type of user> I want to <desired capability>, so that <expected result or benefit>.
Sustainable workflows
The eighth principle of the Agile Manifesto states Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely [Kent Beck, et al. © 2001. Manifesto for Agile Software Development. http://agilemanifesto.org/. Accessed 10 November 2019]. Unfortunately, this objective is not always achieved, but it is the desired goal.
There are several keys to sustainability in iterative development environments. First, the team should never overcommit. They need the skills and resources to do the work required to support the development of the product. They need direct access to customers and users so they can fully assess the feature requirements. The team needs to develop heuristics for estimating the amount of work they can complete in a development iteration. Finally, the team must be in control of stating how much work they can accomplish in any development iteration.
Testing (test-driven and automated)
Testing is an important component of the overall software development life cycle process regardless of whether the development team is following the traditional approach or an agile-based development approach. Testing is a set of quality assurance (QA) activities that confirms the product features meet the business, user, functional, and nonfunctional requirements.
The primary difference between testing practices in the traditional model and those found in agile is the amount of work taken on before testing. As noted in previous sections, the more the development team attempts to implement new functionality before testing, the more likely there is to be an accumulation of bugs and defects that are difficult to locate and resolve. The iterative development nature of agile practices helps reduce the complexity of debugging a software product by limiting the amount of functionality implemented between testing.
Another important aspect of modern agile practices is to develop a test for each piece of code before writing the code. This practice is referred to as test-driven development.
Traditionally, the code is written first and then tested. In the traditional approach, the challenge is to develop a test, after the fact, that accurately supports the original requirement as opposed to writing a test that simply verifies the functionality of their code.
Test-driven development puts the focus on initially creating a test that fails unless the code properly implements the underlying functional requirement. It's not uncommon to have a developer run the test before they write the code to make sure it does fail, thus ensuring there is not a conflict with other code or that other code already fulfills the requirement. Next, the developer creates the code and runs the test again. If the test fails, the developer has not properly implemented the requirement. If the test passes, the developer has confidence they have properly implemented the requirements and that their code works.
Testing environments are another critical success factor in software development regardless of the software development methodology used. Sometimes, organizations take shortcuts due to costs and do not stand up dedicated test environments, forcing the development team to run their tests on their development environments or in the production environments. The problem with running tests on the development environments is that those computing systems likely do not replicate the exact conditions the software must operate in on the production environments. The problem with running tests on the production environments is that a bug could potentially bring the system down.
The only way around the problems associated with testing the production environment is to create a clone of the production environment. Again, modern PaaS offerings make it easier and less costly to stand up a development or test environment to make the conditions found in production environments. Likewise, virtualization makes it possible to set up independent testing environments within a set of clustered servers. Performance testing in test environments helps duplicate the demand loads expected in production environments.
Tools
The introduction of software programming tools came along almost concurrently with the introduction of electronic computing equipment. The primary focus of programming tools is to convert human-readable instructions into machine-readable instructions. The modern instantiation of software programming tools includes text editors and IDEs.
Text editors facilitate code development by providing syntax-aware assistance to code construction based on the type of programming code used by the developer. IDEs provide an integrated suite of tools to assist developers and code creation, software builds, and testing. The suite of tools within an IDE may include text editors, object and data modeling tools, code libraries, compilers or interpreters, debugging tools, and test platforms. IDEs help turn software development into an integrated and automated process and help reduce typing errors and bugs.
In the earliest days of agile, and also used in support of traditional software development practices, efforts were made to simplify business process and information analysis through the introduction of modeling methods and tools. Such tools fell broadly into the categories of RAD and CASE.
CASE tools attempted to reduce the complexity and speed of development by providing a suite of tools to model the physical and logical views of the customer's domain. The most sophisticated CASE tools offered integration capabilities that helped translate the information from one modeling method and tool into another.
The RAD approach, as proposed by its originator, James Martin, added methods and tools for Joint Application Development (JAD). JAD is an approach to capture business requirements through a series of workshops with domain experts and knowledge workers within the customer's organization. As you can probably tell, JAD sessions supported the traditional software development model more than agile, by attempting to collect all requirements at the start of a project. On the other hand, the case tool environment had the goal of expediting development.
RAD tools still exist with an emphasis on modeling and a low-code rapid application development platform. As in the past, the primary focus of RAD tools is a modeling environment that encourages the participation of domain experts in defining and analyzing business and user requirements. However, modern RAD tools also implement capabilities to implement automated workflows and integrated forms with no or minimal amounts of software coding.