In this article, I’ll go over first impressions and experiences of using a low-code tool for UI test automation. If you are unfamiliar with the term “low-code test automation tools”, they are visual tools that allow users to automate tests with little to no programming knowledge.
With the abundance of such tools on the market, it’s not always easy to pick the right one for your specific context. Fortunately, most of these tools will allow you to try them out before you buy them, so you can use trials to compare different tools.
Make sure that you take full advantage of free trials before deciding to commit. The biggest selling point for these tools is a low barrier to entry and rapid test creation.
Our testing context
We wanted a low-code tool to test our new accounting app. This app requires the use of many different tax codes, all of which represent different percentages. And if those percentages aren't calculated correctly, VAT will be incorrect, as will accounting entries.
So we needed to be able to combine many different sets of parameter values across our tests (a technique known as combinatorial testing) and to minimize repetition of any one set of combinations.
We have tried multiple tools from several vendors, such as Katalon, Ranorex, Virtuoso, Testim, testRigor, ACCELQ, and Leapwork.
Debunking misconceptions about low-code test automation tools
Test reliability
Many people claim that tests made with low-code tools aren't as reliable as those powered by tools such as Selenium, Cypress, Playwright, and the like. This was true when the low-code tools first came to market. However, they have matured and have become easier to use.
AI-powered? A little bit.
Many of these tools come with the promise that they are “AI-powered.” Take that with a grain of salt.
For example, many vendors claim that you can author self-healing tests. While this functionality is further along than it used to be, it's still not up to the point where AI is doing all the test maintenance for you. The tools' self-healing capabilities can detect and correct test code in the event of minor UI changes, but your engineers will need to do most of the troubleshooting for failed tests. For now, the claim to offer self-healing tests is mostly marketing.
Our first impressions: several pleasant surprises
Tools that rely on record and playback meet with mixed reactions from people. Those who are comfortable with coding usually prefer custom frameworks because they offer more control and flexibility. But for people without much automation experience, tools of this kind can be a foot in the door to start creating automated tests.
Rich feature sets
After trying out one of these tools for the first time, I was surprised by how feature-rich it was. You could:
- Label your tests
- Set up schedulers
- Run tests locally or on remote grids
- Parameterize your tests
- Copy tests from one project to another
- Edit element locators if the tool didn't automatically pick the best one
Good vendor support
Onboarding for new customers
Onboarding is typically thought through well too, which helps you to hit the ground running. Once we chose a tool, we had several helpful sessions with high-level technical vendor reps.
Dedicated tech support
Dedicated support is one of the biggest advantages of commercial tools. On my project, one of our assertion libraries stopped working. After we contacted the tool's vendor support, the issue was fixed in one or two working days.
Challenges we faced with low-code test automation tools
As mentioned, low-code tools have certain drawbacks, which means you must do your research and test-drive many of these products before deciding which one to use.
We ran into issues, small and large.
- We could create labels for our tests, but we couldn't edit them afterward. (The vendor fixed this issue.)
- When we tried to use a custom npm assertion library, performance slowed to a crawl. (Again, the vendor gave us a fix for this a couple of days after we filed a ticket.)
- Debugging capability tends to be minimal; you can create breakpoints, but that's about it.
- And then there's vendor lock-in. One tool's web UI testing feature set was great, but its mobile UI testing capabilities lagged behind. Had we gone further with that tool, we would have had to use another vendor's tool alongside (or instead of) the first one.
Lack of transparency in pricing
Pricing is not easy to understand at all, especially with many of the big players in this market. Most of the time, you don't even have access to a simple pricing chart or table that tells you how cost varies with the number of concurrent users, available resources, and so on. Often, you will need to talk to a sales rep so they can give you a quote.
In any event, as stated above, make sure to work out some kind of free trial before deciding to buy any tool.
Debugging the output of the tools' black boxes
Debugging can also be very different when compared to custom-coded automation frameworks. You will need to learn to do things the way the vendor intended. This usually doesn’t have a very steep learning curve, but it can still require time to adjust to it.
In essence, visual debugging has its limitations due to the lack of access to the inner workings of the tool. This is not the case when you are using your custom framework, made with Selenium and JUnit, for example.
Limitations when working with multiple browser tabs
I frequently encountered issues when tests required new browser tabs to be opened or multiple browser tabs to be used. All of the record and playback tools I’ve used, so far, have been a little problematic in this regard.
What I saw: I would record a test that opens a page in a new tab. Then I would run it afterwards a few times. After a little while, it would start failing for no apparent reason. Since you are locked into the vendor’s way of doing things, and debugging is a bit limited, you can usually put breakpoints on recorded steps. But it is a rudimentary way of debugging.
A special note if you want to run tests in the vendor's cloud
With a few of the tools we tried, running tests in the vendor's cloud would not work unless we whitelisted the IP addresses of our test environments.
Unreasonable limitations on test step creation and deletion
In one of the tools we tried, it wasn’t possible at all to delete shared steps (steps that can be reused across multiple tests to minimize repetition). Even if no test was using a shared step, it couldn’t be deleted, only hidden, which didn’t make much sense.
Mediocre performance and limited control of the test player
The test players were sluggish at times, although the vendor was able to resolve some issues connected to that. Also, we found that we couldn't always configure the player to skip tests. The best that could be done was to put the tests to be skipped under quarantine.
Are these tools really "low-code"? only to a point
All these tools come with a promise that they can be used with zero programming knowledge. But, across the board, any configuration that's just a little more advanced requires you to roll up your sleeves and write some custom code. For example, the tool sometimes wouldn't allow me to specify the DOM element to target, so I had to write custom JavaScript in those cases.
To wrap up
After we tried several tools, we scored them based on how they performed in different areas:
- Documentation
- Ease of use locally and via the web
- Customization capabilities
- Parameterization capability
- Pricing
In the end, we chose the tool with the highest overall score.
Commercial low-code test automation tools have evolved significantly, especially over the last decade. They are becoming more and more user-friendly, allowing fast test creation, reducing development time, and helping to improve collaboration. For simpler UI test automation, these tools can be huge time savers.
- Integration with other tools: As these tools have been continually developing, we now have many options to integrate these platforms with other tools we’re using. For example, when a test fails, a bug can automatically be created in Jira, or the test runner can send a Slack message when a test run is complete, and more.
- Test maintenance: Maintenance of tests created by record and playback tools has also become easier, but it’s still a bit more repetitive and time-consuming compared to custom frameworks.
- Limited flexibility, scalability, and customisation: On the downside, these proprietary tools come with limited flexibility and customisation, especially if your automation needs are very complex and demanding. You might also experience issues with scaling. As tests grow in number and complexity, maintenance and execution times can also increase considerably.
- My recommendations for optimal use of low-code test automation tools: For relatively straightforward apps (especially web), a lot of these tools will do a very decent job. In the end, you should take time to analyze your exact automation needs, try out several low-code tools, and go with the one that works best for you.
For more information
- How Can You Contribute to Automation If You Don't Know How to Write Code?, Luke Liu
- Jumpstarting Your API Test Automation: Which Test Creation Tool Is Easiest To Learn?, Ejob Gaius
Don’t Be a Fool With a Tool ‒ Fundamental Knowledge for Proper Test Automation, Christian Baumann
