Introduction to mobile test automation
Before we start learning about mobile test automation, let's understand what functional test automation is.
Test automation has always been a fundamental part of the software testing lifecycle for any project. Organizations invariably look to automate the repetitive testing actions in order to utilize the manual effort thus saved for more dynamic and productive tasks. Use of automation tools also allows utilization of system idle time more effectively. To address these needs, there are a plethora of tools available in the market along with various frameworks and implementation techniques. There are both open source and licensed tools available in the market. Tools such as HP's Unified Functional Testing (UFT), formerly known as QuickTest Professional (QTP), TestComplete, Selenium, eggPlant, Ranorex, SilkTest, IBM Functional tester, and numerous others, provide various capabilities for functional automation.
However, almost all of these tools are designed to support only a single operating system (predominantly Windows—owing to its popularity and the coverage it enjoys across industry verticals), although a few provide support for other lesser-used operating systems, such as Unix, Linux, Sun Solaris, and Apple Macintosh.
As far as functional automation is concerned, you don't need to even consider the implications of supporting multiple operating systems in most cases. With Windows as the only operating system that is supported, there aren't any considerations for different operating systems. If the application is a web application, then there may be a need to do cross-browser testing, that is, testing automation on various browser types (Chrome, Firefox, and Safari besides Internet Explorer) and their respective versions.
Also, as far as functional automation is considered, there is a very clear demarcation between nonfunctional and functional requirements. So, an automated solution for functional testing is not required to consider factors such as how others processes running on the machine would impact it, or any of the hardware aspects, such as the screen resolution of monitors and the make of the machines (IBM, Lenovo, and others).
When it comes to mobile automation, there is an impact on the test suite design due to various other aspects, such as operating systems (Android, iOS, Blackberry, Windows) on which the application is supposed to be accessed, the mode of access (Wi-Fi, 3G, LTE, and so on), the form factor of the devices (tablets, phones, phablets, and so on), and the behavior of the application in various orientation modes (portrait, landscape, and so on).
So, apart from normal automation challenges, a robust mobile automation suite should be able to address all these challenges in a reliable way. In the coming chapters, we will learn about how to create such frameworks, but before that, we need to have a thorough understanding of the challenges.
Fragmentation of the mobile ecosystem is an aspect that compounds this manifold problem. An application should be able to service different operating systems and their flavors provided by original equipment manufacturers (OEMs), such as Apple with iOS, Google's Android with Samsung, HTC, Xiaomi, and numerous others, Windows with Nokia and HTC, and even Blackberry and other lesser-used operating systems and devices. Add to this the complexity of dealing with various form factors, such as phones, tablets, phablets, and their various hybrids.
The following figure is a visualization of the Android market fragmentation over various equipment manufacturers, form factors, and OS versions:
As we know, test automation is the use of software to automate and control the setting up of test preconditions, execution of tests, test control, and test reporting functions with minimum, or ideally zero, user intervention. Automating the testing for any mobile application is the best way to ensure quality, and to achieve the quick and precise results that are needed to accommodate fast development cycles.
Organizations look toward functional test automation primarily to reduce the total cost of ownership over a period of time, and to ensure the quality of the product or application being developed. These advantages are compounded many times for mobile test automation and hence it provides the same advantages, but to a much greater degree.
The following are the various advantages of mobile test automation for any project:
- Improved testing efficiency: The same scripts can be used to run uniform validations across different devices, operating systems, and application types (of the same application), thereby reducing the test execution effort considerably. This also means that the return on investment (RoI), which typically takes about 3-5 cycles of executing the conventional functional automation to achieve breakeven, is viable in most cases within the first release itself, as mobile testing is typically repeated on many devices. So, in this case, fragmentation acts as a positive factor if the automation is employed properly, whereas, with pure manual testing, it greatly increases the costs.
- Consistent and repeatable testing process: Human beings tend to get bored with repetitive tasks and this makes such a test prone to errors. Due to the effect of fragmentation in the mobile world, the same application functionality needs to be validated across various combinations of operating systems, application types, device manufacturers, network conditions, and many more. Hence, the use of automation, which is basically a program, ensures that the same scripts run without any modifications every time.
- Improved regression testing coverage: The use of automation scripts allows the regression tests to be iterated over multiple combinations of test data. Such data-driven scripts allow the same flow to be validated against different test data combinations. For example, if an application allows users to search for the nearest ATMs in a given area, basically, the same flow would need to be tested with various zip codes as inputs. Hence, the use of automated scripts would instantly allow the test coverage to be increased dramatically.
- More tests can be run in less time: Since automated scripts can be run in parallel over various devices, the same amount of testing can be compacted inside a much smaller time window in comparison to the manually executed functional testing. With the use of automation scripts that include device setups as preconditions, the execution window can be exponentially reduced, which otherwise would take a manual tester considerable time to complete.
- 24/7 operation: Although any functional automation suite can lead to better resource utilization in terms of executing more number of scripts in lesser time, with respect to mobile automation, the resources are often expensive mobile devices. If functional testing is done manually, then more of the same devices need to be procured to allow manual testers to carry out tests, and especially, more so in the case of geographically distributed testing teams. Mobile automation scripts, on the other hand, can be triggered remotely and can run unattended, reducing the overall cost of ownership and allowing 24/7 utilization of devices and tools.
- Human resources are free to perform advanced manual tests: Having automation scripts perform repetitive regression testing tasks frees up the bandwidth of manual testing teams for exploratory tests that are expensive to automate and cumbersome to manage. Hence, the use of automation leads to a balanced approach, where testers can perform more meaningful work and thereby improve the quality of delivered applications. In mobiles, since regression is more repetitive on account of the fragmentation problem, the amount of effort saved is manifold, and hence, testers can generally focus on testing aspects such as user interface (UI) testing and user experience testing.
- Simple reproduction of found defects: Since automation scripts can be executed multiple times on demand and are usually accompanied with reports and screenshots, defect triangulation is easy and is just a matter of re-execution of automation scripts. With pure manual testing, a tester would have to spend effort on manually recreating the defect, capturing all the required details, and then reporting it for defect tracking. With mobile automation, the same flow can be triggered multiple times on a multitude of devices hence, the same defect can be replicated and isolated if it occurs only on a specific set of devices.
- Accurate and realistic real-life mobile scenarios: Since a mobile requires tests to be specifically designed for variable network conditions and other considerations, such as device screen sizes, orientation, and more, which are difficult to recreate accurately with pure manual testing effort, automation scripts can be developed that accurately to recreate these real-world scenarios in a reliable way. These types of tests are mainly not required to be developed for functional automation suites, and hence, this is one of the major differences.
For the most realistic results, conventional wisdom is to test automation on actual devices—without optical recognition, emulation, jailbreaking, or tethering. It is impractical to try to automate everything, especially for mobile devices. However, leveraging commercial off-the-shelf (COTS) tools can vastly reduce the cost of automation and thereby enhance the benefits of the automation process.
In the following section, we will discuss in detail the challenges that make mobile automation vastly different from conventional functional automation.
The following are some of the issues that make the effective testing automation of mobile applications challenging:
In subsequent chapters, we will discuss how to effectively manage these using various techniques.
Free Chapter
