This is a guest post by Peter G Walen.
Test Automation has become ubiquitous, a bit like wifi. It is everywhere in software development. Sometimes it is done powerfully and well, other times, it is suboptimal. Let’s revisit some of the reasons why.
“Automated Testing,” “Machine Assisted Testing,” “Computer-Aided Testing.” Pick a term. Pundits have bandied them about, in sometimes pedantic rants, for many years. They seem to get used more and more often as time goes on.
The basic premise is that Automated Tests, and Automated Testing, test software faster, more often and speed accurate delivery. From my experience, this can be the case. The problem is, I have seen not anything guarantee that.
While I’ve written about test automation a bit for TestRail (see here, and here), and have spoken on related topics in the past, maybe it is time to again ask the question “Why do we need automated tests, anyway?”
One basic set of responses are above. Good test automation can run tests quickly and predictably. This will avoid situations where a person executing the steps by hand might miss a step, or enter a value that is not intended as part of the test. This has been the promise since some of the earliest “automation” tools that focused on record and playback of the UI.
Another is that organizations can include tests in the build process and can be initiated each time a build completes. This could be a form of CI or a more fundamental tool. Something that can be initiated when the build process and CI tests complete, that can start scripts created for the project being checked in.
A third example, and one the last example above depends on, is that of automated functional tests. These can be created by the development team while the application software is being developed and are used to exercise the change in their local environment. This differs from TDD in that these tests are not part of the development process per se but developed in parallel with the application.
Each of these ideas is good. Together they present solid ideas for test automation. The focus is on definite goals that can show measurable, repeatable behavior and activity trends.
Challenges come when we look beyond the high-level goals and intent. There is a consistent effort in some circles, and by some practice vendors, to associate Agile software development with automated testing, and only automated testing. This leads to a distortion of the above-stated desires and goals.
Such challenges are often inserted or created by people with a shallow understanding of testing, test automation or testing tools. Some are blatantly presented as “facts” and “proven outcomes” by some unscrupulous tool vendor representatives.
Challenge 1: Automation makes everything faster
This is an interesting challenge. It often comes from a belief that Agile and/or SCRUM will always make delivery faster and the quality better. What makes this interesting is while those things may be true sometimes, it is not true in all absolutely all instances.
Test automation can make some things faster, sometimes. However, there is no certainty that total time to test will be reduced, nor that it will decrease time to delivery. There is no certainty that automation by itself will do anything faster, other than initiate consecutive executions.
Under the right circumstances, building test automation code while the application code is being built can help let testing start as early as possible. Building both the test and application code together, in parallel, can help the people developing both work together to make sure there are fewer unexpected surprises in calls, interfaces, log entries, etc., when it comes to time to test.
Challenge 2: Test Automation means you can do better testing with fewer people
This challenge is brought to us by the people keeping accounts and budgets. Because the organization is spending “so much money” the obvious “cost savings” comes in reducing headcount. That is, people. The obvious people in question to be reduced are usually the people who do “manual testing.”
After all, automated testing will make it so we won’t need anyone doing manual testing. Because if we don’t get rid of “overhead” (people) the organization will have a cost increase, not break-even and not a “savings.”
The issue is, yes, we won’t need people pounding away at manual scripts, as long as it makes sense to automate those scripts. However, will the people who remain understand enough about the product to help design good tests, beyond simple unit tests? Will they be able to design full functional tests? And what about maintaining the integrity of the regression suite? Not just the coded scripts, but the logic behind them?
If so, then this might be true. However, it is more common to see the questions about understanding the system or application under test and fully understanding the automation code around them are often for many reasons, mutually exclusive. Thus, to maintain some rigor in testing, an organization might not be able to reduce the number of people by the expected amounts.
Challenge 3: One tool will fit all your needs
This challenge, perhaps more than the others, causes much consternation among software development teams.
The “best tools available” can do some amazing things. Still, I’ve not seen a hammer or screwdriver trim a piece of 4×4 as needed. A hammer might sink a screw, not very well of course, but it could do it. Likewise, a screwdriver could, maybe, drive a nail. I suspect it would take a bit of doing and massive amounts of frustration, but one could, conceivably, get it done.
Why would you want to?
Similarly, why would you want to use a test automation tool for functions it was never intended or designed to be used for? Some tools are very, very good for performance tests. Some are very good for API level tests. Some are very good for UI tests.
Some of these tools might have secondary uses to broaden their reach. The challenge comes in learning them, learning them well enough to teach others, and mastering it well enough to do what it was promised to do.
Why do we need test automation?
This brings us, finally, to the question. Why do we need test automation? If the things that commonly are said it does for us may not actually do what we think, why do we need it?
Let us consider tools, in general. Why do they exist? Do we need them? What do they do for us?
We can use our hands to dig in the soil, say to plant seeds or transplant a seedling. It might be more efficient to use a small trowel. If we need to dig a larger hole, a spade might be a better choice. A shovel could be used to dig an even bigger hole and shift small and medium-sized rocks as needed. A powered tool, such as a back end loader, could dig a much bigger hole, much faster than we could with any of those other tools.
Why use them?
For some jobs, shifting soil with the fingers is just fine. For bigger jobs, tools make sense. Why use those tools? That is simple. They allow us to do the job quickly and get on to other tasks.
Test automation tools serve the same purpose. Test automation, and automated testing, allow us to be freed of the mundane, hole-digging tasks of testing, so we can get on to the other tasks requiring deep, considered thought.
Peter G. Walen has over 25 years of experience in software development, testing, and agile practices. He works hard to help teams understand how their software works and interacts with other software and the people using it. He is a member of the Agile Alliance, the Scrum Alliance and the American Society for Quality (ASQ) and an active participant in software meetups and frequent conference speaker.