When and how should QA testing for games be performed?
There are lots of discussions about when QA is executed. Traditionally, QA was always done toward the end of development. This was especially true when using a waterfall system. Following the waterfall development rules, QA testing is performed only after development has concluded and the finished software is handed over to the testing team. The following figure shows how the waterfall model follows specific steps in a particular order:
Figure 1.1 – Waterfall system
In a waterfall system, testing is done only when development is finalized. However, with the rise of agile methodology, gaming studios have gradually started to adopt this method of development as well. Furthermore, with the explosion of mobile gaming and F2P games, it became obvious that agile development was necessary for efficient live ops. This required that we change the way we test games. Handing the product to the testing team when “done” became impossible, because modern F2P games never end, and hence development is never finished.
When should QA for games be performed?
Game development has evolved into an iterative model, where we develop smaller chunks of code that are tested as they are developed. As we can see in the following figure, agile development is iterative – small, usable pieces of code are frequently released and tested:
Figure 1.2 – Iterative development
The aim of agile methodology is to develop minimum viable features that can be continuously improved in future iterations.
There are several unique aspects of testing in an agile environment:
- Testing is continuous: a QA tester still holds the position of gatekeeper and has the responsibility of validating whether a feature or the whole build is of good-enough-quality to be released to the target audience. However, besides that, QA testers also test early features, validate usability flow, develop test cases based on use case scenarios, and perform many other tasks. If your game is live, the QA team will additionally handle live testing and verification of bugs coming from player support, forums, and your game’s social media channels.
- To be able to support continuous testing, QA is part of the agile team, rather than a separate unit. It is impossible to get full insight into agile development if QA is not part of the team. In modern game development, we still often use outsourced testers for the QA-heavy stages: first-time releases, major updates, or significant changes such as the introduction of multiplayer. However, very often we have situations where we have one full-time QA team member and a scalable external QA partner.
- Testers must adapt their approach to testing based on the development stage and feature readiness. It doesn’t help to report minor bugs while testing the first version of a major feature. This can be particularly challenging for QA testers. These professionals need to have experience, skills, and great collaboration with the rest of the team to fully understand the product priorities and focus their testing effort on the right thing at the right time. In the following table, you can find a high-level overview of what the testing focus should be in each stage of game development.
MILESTONE
TESTING FOCUS
HOW TO TEST
PRE-PRODUCTION
User stories, scalability, game design flow
Document reviews, decision tables
ALPHA
Core gameplay, fun factor, stability
On the selected principal device, emulators
BETA
UI, UX, game flow, FTUE, monetization, coverage
On a wide range of target devices
RELEASE CANDIDATE
Balancing, meta game, localization, game polish, shop and monetization, coverage
On the main supported devices
Table 1.1 – Testing focus per development phase
- Testers represent the player in the team. It’s their job not only to test the code, but also to collaborate in the development process, putting themselves in the players’ shoes. This approach requires testers to have an in-depth understanding of the target audience, its preferences, its likes and dislikes, and a solid knowledge of competitors’ games. Most of the time, the QA specialist closely collaborates and communicates with player support; testers can consequently gain precious insights into players’ gaming patterns, game preferences, and the things they find most frustrating.
Based on everything we have discussed up to here, we can freely conclude that optimally, QA testing should start as soon as we start development. Many studios still hold on to old ways of working and test only at the end of the development cycle, as it’s perceived to be cheaper. This type of testing is usually very stressful as there is rarely enough time to ensure adequate testing coverage and many serious bugs can easily creep into the live game. By having QA performed in a more iterative way, we decrease that risk. This approach is potentially much cheaper than having to fix the damage of a failed product launch.
By now, we have a better understanding of when games should be tested. But, how should the testing be undertaken?
How should QA testing for games be performed?
When we start thinking about testing, it’s important to have some kind of understanding of what the desired outcome of our test is. That will allow us to compare the actual result of the test with the desired outcome. For some tests, this might be obvious – for example, the test case open settings and mute the sound has the obvious desired outcome that the game is muted.
For other scenarios, however, it’s not so simple. If we investigate a test case such as check whether game loads, it might seem deceivingly obvious: of course, the game should load! But, how long should the loading process take? Should there be any sort of loading indicator? Will the audio be on or off while loading? As we can see, it’s not necessarily that simple to determine what the desired outcome is. This test case is something my team and I had to experience ourselves to find the answer to. It took our team almost a week, while also researching competitors, to finally agree on what the expected outcome of that particular test case should be. The desired outcome of a test case is also sometimes called the test oracle.
More often than not, we also use test cases for testing. Even if they are not a mandatory part of game testing, they are very frequently used across various gaming studios. We will study test cases in depth later in the book, but for now, it’s sufficient to become familiar with their general definition. Test cases are written instructions that list specific steps for how testing should be performed.
Besides test cases, testers can execute tests using different types of documentation. Many teams create their own product test plans, which are detailed, carefully planned documents describing all aspects of testing that will be undertaken for a specific game. Test plans usually don’t contain test cases themselves, but they specify test case tools, who created them, how, when, and based on what. The plan might also contain links to the test case repository.
If your team has fully embraced agile development practices, you might also use test charters. Test charters are less formal documents compared to test cases. They generally contain information about what the testing goal is and some ideas and approaches regarding how to reach it.
Lastly, let’s talk about who is doing the testing. By now, you have probably realized that we are using tester and QA specialist/tester as interchangeable terms. There are really no differences between the two, but, as we don’t have a unified industry terminology, there might be some differences between the two, depending on how a particular studio interprets testing tasks. However, besides professional QA personnel, there are other people who perform testing as well:
- Developers
- Members of your development team apart from developers (designers, artists, etc.)
- The marketing team
- Product managers
- Players
It’s part of the developer’s job to do testing as well, although they use a very different methodology from testers. Developer testing focuses on what is called white-box testing. This implies testing mostly focused on unit tests and code reviews. While these tests are exceptionally useful, they have slightly different purposes compared to the tests performed by QA teams.
In modern agile teams, most of the team performs at least some type of testing. Designers review the logic and validity of their designs. Artists check how their work looks and feels in the game. UX designers review any changes in the user flow. The producer will have an overall look at all the components, trying to assess the level of completion and early build quality. All of these help the QA team to get a better picture of where the specific risks are and focus their efforts in the right place at the right time.
When we talk about marketing, we primarily talk about player support. It is wise to keep the player support team informed well in advance about any game developments and provide them with design documents and access to early builds. That will help them understand the game better and address customer comments and complaints properly. As they are the people spending the most time with the players, very often, you get valuable feedback from those early play sessions.
Lastly, players themselves are also testers, whether we like it or not. They will play the game in all the possible ways they see fit and report bugs through player support, forums, or social media. You might ask, if they test it themselves, why would we bother paying specialists to do it? This is because, usually, when a player notices bugs in the game, it’s already too late, especially if the bug is a serious one. We might already have lost part of our players or a significant amount of revenue.
However, there are ways of utilizing players’ testing in a more formal way. We do that through beta testing or playtests. In these types of tests, we allow smaller, often specifically selected groups of players to play early builds of the game. The primary goal of these tests is to get an idea of how players will receive the game and their likes and dislikes, but they also very often yield useful bugs.