Test Automation: Always The Approach Of First Resort?

By Eduardo Fischer

It’s no surprise to anyone that automation testing has changed the world of software testing. However, even though automation solves some problems, its availability also raises questions that we need to answer if we want to move forward with our testing plans. 

Perhaps the most important question to answer is whether to do our test automation first or to work first on manual test planning. Certainly there are pros and cons and not every strategy is going to be best in all cases. This article will look at some considerations that will help you make the decision. 

When “Automate First” May Be Better

If there are too many test cases to cover manually, it’s a good sign you should automate first. I once worked for a logistics startup with a particular problem: precise shipping quotes were required for e-commerce shops. It was obvious from the start that effective manual testing was improbable. There were simply too many combinations to list them all: postal codes, product weights, sizes, insurance values, special rules that changed for every shipping company, and so on. Testing by hand would take me days or weeks to produce valuable testing results. I decided to automate my testing and forego manual testing entirely.

Because I’d decided early to automate my testing, I soon was able to execute hundreds of tests in minutes. But I still could not automate every possible scenario. I focused on price and time to ship as the primary values to vary, and then I used boundary value analysis and equivalence partitioning to focus on the most typical or error-prone scenarios.. 

The strategy was a success. We found many bugs before they went into production, and the automated tests allowed for quick testing of bug fixes. Any time there was an update to the pricing, the tests could be updated and executed in a reasonable time.This simply would not have beene possible with manual testing.

When To Focus On Manual Testing

If you’re a developer, chances are you love to code. This can mean that you also find excuses to automate processes even when it is not necessary. Maybe there is a new framework or language you want to try out.  However, the truth is that you should automate only when you need to, not when you simply want to. Even if we are hired as test automation engineers or specialists we should think long and hard about what to automate and when before we write a line of test code.

First of all: if you don’t have a need for regression testing, it is usually faster to test manually one time. Remember that we are hired to solve problems, not simply to code or automate. Automation is a means to an end and so is manual testing. If you are going to be using a minor feature just for one day, is it really necessary to have a complete suite of tests just for that one day? Yes, it could mean fewer bugs the day after, but did that small reduction in bugs justify the cost of writing the code?

Similarly, you should not waste a lot of automation time on features that are not going to be updated. Web and mobile apps can be updated to new versions fast and this often calls for automated regression tests. But think about embedded systems like microwaves, rice cookers, elevators, cars, or any hardware system without an internet connection. Once they are out on the market for sale there is no chance to update them easily. If there are no further updates, all test automation has reached the end of its utility, and it’s likely that it was not cost-effective to implement.

When There Is No Easy Answer

A few scenarios will not present clear answers. For example: frequently updated software is prone to new bugs or the recurrence of old ones. We can use automated regression tests to save ourselves some time. But what if code changes so fast that we have no time to update our regression test suite?

Good automation can accommodate rapidly changing code, to a point. But if we as testers can’t keep up with the changes it’s time to think about new strategies. Often the solution to this problem is a mixed approach. Automation can cover core aspects of the project, those that you are certain won’t be changed frequently. For the code that is not covered by automation, execute manual testing whenever there are changes. And sometimes there are no easy solutions.

As humans of course we have our favorite methods of working. However, more important than our own preferences is to take a rational approach to what is potentially the best way to work on the problem. Certainly we need to take into account the skills of our team, the number of people on the team, their available time, and so on.

In the end there is no solution that is the best for all cases, but with the right mindset we can at least find a good solution for the case in front of us at the moment.   

Further Reading

Great Test Automation Does Not Absolve You from Manual/Exploratory Testing, Marit van Dijk

Devs Should Love Your Tests, Carlos Kidman

Author Bio

Eduardo Fischer is a quality assurance engineer who focuses on API and UI test automation and is currently working for an international stockbroker. He discovered his passion for testing during his time as a back-end developer and eventually made testing his fulltime job. You can find him on LinkedIn: https://www.linkedin.com/mwlite/in/eduardo-fischer-dos-santos