Why automated testing?
Plainly said: it all boils down to saving you a lot of hassle. There are no emotions or time-intensive execution that will keep you from (re)running tests. It's just a matter of pushing the button and getting the tests carried out. This is easily reproducible. Each time you push the button, the test will be executed exactly the same. It is fast to execute, as automated tests are "light-years" faster than any of us could do, and objective: a test fails or succeeds, and this will be part of the overall reporting. No human emotions that could hide it away.
If it was as straightforward as that, then why haven't we been doing this in the Dynamics 365 Business Central world all these years? You probably can come up with a relevant number of arguments yourself. Of which the most prominent might be we do not have time for that. Or maybe: who's going to pay for that?
Why not?
Before elaborating on any arguments, pros or cons, let me create a more complete why not? list. Let's call it the whys of non-automated testing:
- Costs are too high and will make us uncompetitive.
- We are not used to doing it this way and our sales and management team don't see the benefits and cannot be "convinced."
- Dynamics 365 Business Central platform does not enable it.
- Customers do the testing, so why should we bother?
- Who's going to write the test code? We already have a hard time finding people.
- Our everyday business does not leave room to add a new discipline.
- There are too many different projects to allow for test automation.
- Microsoft has tests automated, and still, Dynamics 365 Business Central is not bug-free.
Why yes?
Dynamics 365 Business Central is not as simple as it used to be way back when it was called Navigator, Navision Financials, or Microsoft Business Solutions – Navision. And the world around us isn't as one-fold either. Our development practices are becoming more formal, and, with this, the call for testing automation is pressing on us for almost the same reasons as the whys of non-automated testing:
- Drive testing upstream and save costs.
- Dynamics 365 Business Central platform enables test automation.
- Relying on customers to do the testing isn't a great idea.
- Having a hard time finding people – start automating your tests.
- Test automation will free up time for everyday business.
- Keep on handling different projects because of test automation.
Let's check each of these aspects separately.
Drive testing upstream and save costs
Regarding costs, I tend to say that a Dynamics 365 Business Central project goes 25% over budget on average, mainly due to bug fixing after go-live. Quite a number of attendees of my workshops, however, have tended to correct me, saying that this is on the low side. Whatever percentage is the case, the math is quite simple. If you're spending 25% extra at the end of the line, why not push it upstream and spend it during the development phase on automated testing, and meanwhile build up reusable collateral? Even more, this also ends up being a powerful multiplier as any defect solved earlier in the project chain, or what we also call upstream, is a factor cheaper than being solved later in the project, that is, downstream.
If my memory is not failing me, this factor could be as large as 1,000. During my time at Microsoft in the 2000s, research was performed on the cost of catching a bug in the different stages of developing a major release of a product. Catching a bug after the release was found to be approximately 1,000 times costlier than when catching the bug already at requirement specification.
Translating this to the world of an independent software vendor (ISV), this factor might roughly be 10 times lower. Meaning that the cost of catching a bug all the way downstream would be 100 times higher than all the way upstream. In the case of a value-added reseller (VAR) doing one-off projects, this factor probably is 100 times lower. So, whatever the factors would be, any spending upstream is more cost-effective than downstream, be it more formalized testing, better app coding, code inspection, or specifying more detailed requirements.
Note
A while ago, I wrote a joint blog post that also discussed this. You might want to read it:
https://www.fluxxus.nl/index.php/BC/from-a-testing-mindset-to-a-quality-assurance-based-mindset.
Dynamics 365 Business Central platform enables test automation
In all honesty, this is a no-brainer as this is the topic of this book. But it is worthwhile realizing that the testability framework inside the platform has been there ever since version 2009 SP1, released in the summer of 2009. So, for over 10 years, the platform has enabled us to build automated tests. Does it sound strange if I say that we have been sleeping for a long time? Sticking to old habits? At least most of us.
Relying on customers to do the testing isn't a great idea
I agree that customers might know their features the best, and as such, they are the presumable testers. But can you rely 100% that testing isn't being squeezed between deadlines of implementation, and, in addition, between deadlines of their daily work? And still, in what way does their testing contribute to a more effective test effort in the future? How structured and reproducible will it be?
Posing these questions answers them already. It isn't a great idea, in general, to rely on customer testing if you want to improve your development practices. Having said that, this doesn't mean that customers should not be included; by all means, incorporate them into setting up your automated tests. We will elaborate on that more later.
Having a hard time finding people – start automating your tests
At this very moment, all implementation partners in the Dynamics world are having a hard time finding people to add to their teams in order to get the work done. Note that I deliberately didn't use the adjective right in that sentence. We all are facing this challenge. And, even if human resources were abundant, practices show that, with a growing business in either volume or size, the number of resources used does not grow proportionally.
Consequently, we must all invest in changing our daily practices, which very often leads to automation, using, for example, PowerShell to automate administrative tasks and RapidStart methodology for configuring new companies. Likewise, writing automated tests to make your test efforts easier and faster. Indeed, it needs a certain investment to get it up and running, but it will save you time in the end.
Test automation will free up time for everyday business
Similar to getting the job done with proportionally fewer resources, test automation will eventually be of help in freeing up time for everyday business. This comes with an initial price but will pay off in due time.
Note
A logical question posed when I touch on the topic of spending time when introducing test automation concerns the ratio of time spent on application and test coding. Typically, in the Microsoft Dynamics 365 Business Central team, this is a 1:1 ratio. Meaning that, for each hour of application coding, there is 1 hour of test coding.
Keep on handling different projects because of test automation
Traditional Dynamics 365 Business Central implementation partners typically have their hands full of customers with a one-off solution. And, due to this, they have dedicated teams or consultants taking care of these installations; testing is handled in close cooperation with the end user, with every test run putting a significant load on those involved. Imagine what it would mean to have automated test collateral in place – how you would be able to keep on servicing those one-off projects as your business expands.
On any major development platform, such as Visual Studio, it has been a common practice for a long time already that applications are delivered with test automation in place. Note that more and more customers are aware of these practices. And, more and more will ask you why you are not providing automated tests for their solution.
Each existing project is a threshold to take having a lot of functionality and no automated tests. In a lot of cases, a major part of the features used is standard Dynamics 365 Business Central functionality, and, for these, Microsoft has made their own tests available since version 11, that is, Dynamics NAV 2016. Altogether, over 33,000 tests have been made available for the latest version of Business Central. This is a humongous number, of which you might take advantage and make a relatively quick start on. We will discuss these tests and how you could get them running on any solution later.
Some more arguments
Still not convinced why you would/should/could start using test automation? Do you need more arguments to sell it inside your company and to your customers?
Here are some more arguments:
- Nobody loves testing.
- Reduced risks and greater satisfaction.
- Once the learning curve is over, it will often be quicker than manual testing.
- Leveraged development practice.
- Shorter update cycles and the AppSource paradigm.
- Test automation is required(?).
Nobody loves testing
Well, almost nobody. And, surely, when testing means rerunning and rerunning today, tomorrow, and next year, it tends to become a nuisance, which deteriorates the testing discipline. Automate tasks that bore people and free up time for more relevant testing where manual work makes a difference.
Reduced risks and greater satisfaction
Having automated test collateral enables you to have quicker insight than ever before into the state of the code. And, at the same time, when building up this collection, the regression of bugs and the insertion of new ones will be much lower than ever before. This all leads to reduced risks and greater customer satisfaction, and your management will love this.
Once the learning curve is over, it will often be quicker than manual testing
Learning this new skill of automating your tests will take a while, no doubt about that. But, once mastered, conceiving and creating them will often be much quicker than doing the thing manually as tests often are variations of each other. Copying and pasting with code is ... well ... can you do such a thing with manual testing? Not to mention to choose between re-running these automated tests or the manual tests.
Leveraged development practice
Test automation is not just automating your test practice and speeding up some parts of your daily work, it's opening up new possibilities, as obvious as they can be. Just to mention a few:
- Not having to manually (re)execute the obvious, ever-reappearing tests opens up room for manual testers to focus on challenging the system, to really try to break the damn thing.
- As automated tests can be executed by anybody or any automated process, a developer has a new tool at hand while developing the application code with fast feedback: with any change relevant, automated tests can be run to check whether no regression occurs.
- Test automation needs more details upfront, which will take some more time, but at the same time allows a much better, more meaningful discussion about the requirements; through the test definitions, the requirements are better tested.
Note
You might want to read my blog post for more examples of how your practice can be leveraged: https://www.opgona.training/index.php/2021/02/25/test-automation-its-not-a-buzz-word-but-a-grand-helping-hand/.
Shorter update cycles and the AppSource paradigm
With iterative methodologies – such as Scrum and Agile – and cloud services, it has become a normal practice that updates are delivered with shorter time intervals, leaving even less time to get full manual testing done, by customers and dedicated resources in your team. Clearly, Dynamics 365 Business Central is a part of this story and your app too. As it will also be a part of the AppSource paradigm, where companies around the world use Business Central in the cloud and are able to basically install any app from AppSource and start using it without any formal relation to and instruction from the manufacturer. If the app appears to be buggy, then they probably will just as easily uninstall it and go for another. You cannot afford for your app to be put aside because your testing practice does not fit this reality. In the Software as a service (SaaS) world, automated testing is a must to guarantee a quality product.
Test automation is required(?)
In the first edition, I started this section with:
This is no longer the case as Microsoft does not require automated testing as part of the AppSource validation anymore. They, however, strongly recommend … using automated tests, hence the question mark added to the section heading.
But even though Microsoft isn't forcing us anymore, know that our customers will be requesting this from us more and more, seeing that your competitors do practice and provide automated testing. The latter giving your competitor the advantage of having more time to add features and release more often.
Note
Read Krzysztof Bialowas' blog post where he discusses this Microsoft policy change with a reference to the Microsoft communication on this:
http://www.mynavblog.com/2021/02/16/automated-tests-dont-listen-to-microsoft/.
Silver bullet?
Notwithstanding all the above, you might rightfully wonder whether test automation is a silver bullet that will solve everything. And I cannot deny that! However, I can tell you that, if exercised well, it will surely raise the quality of your development effort. As mentioned before, it has the following benefits:
- Easy to reproduce
- Fast to execute
- Objective