The maintenance problem
So far, we’ve pointed out that repeatable/repeating tests create less coverage than changing the approach for each build. We did this through a metaphor and didn’t spend a great deal of time on how to adjust the testing to increase coverage over time. Computer-driven tests are the ultimate example of this. Humans following documented steps run this risk of missing ideas to explore, of asking what if…?. With computers, it’s guaranteed because of the fulfillable promise of repeating tests.
So, we created a simulation that repeated tests and, surprise surprise, it’s worse at finding bugs.
There are a couple of additional challenges that do not appear in the simulations.
First, there is what the test is trying to do. Consider a test like this, which Matt created in Selenium IDE, an open source tool that records and plays back tests. All the test does is go to Excelon Development’s main page, then click on Consulting...