How to investigate and resolve test failures.
In this post I’m going to demonstrate using the debugging features in Selenium IDE to investigate and fix failures in your automated tests.
As you’ve probably already seen, Selenium IDE reports failures in the log console, and failed test cases within the suite are shown in red. This can be a pain if you’re used to seeing lots of green and suddenly you have to find out why all your tests have turned on you!
The failing test cases and commands are highlighted in red, so it’s easy to see what failed, but quite often it can be difficult to understand why it failed. This information is all in the log, but if you’ve run more than just a couple of tests then it will be difficult to match up the failing commands to the errors in the log.
There are a few ways to debug these failures and work out if your application under test has bugs, or perhaps your tests require some changes.
Execute Individual Commands
When you double-click a command in Selenium IDE it will be executed – this is very handy when first writing your tests. You could use this to step through each command in each test case until you find a failure, and investigate from there.
Custom Log Messages
There’s a command in Selenium IDE named
echo and this will basically repeat whatever you type into the log when the test is executed. You could use this to output a message associated with an expected failure, and then find this in the log. You do still have to scroll through the log to look for your messages though.
You can use the speed slider to slow down how fast your test commands are executed. This can be helpful as you can watch the application under test while your tests are running and see any obvious issues that your tests are failing on. You can also use this in combination with executing commands manually by pausing your tests.
Pause, Break and Step
You can pause your test at any time, either by clicking the Pause icon in the toolbar or by setting breakpoints. Set a breakpoint by right-clicking on a command you want Selenium IDE to pause before executing, and selecting ‘Toggle Breakpoint’ from the context-menu. You will see a small pause icon appear to the left of your command.
When Selenium IDE is paused, the Step icon becomes available. You can use this to execute one command at a time, which can be very useful when approaching a failure or working through a number of verification failures. If you have set multiple breakpoints then you can click the Resume icon to continue executing your tests and stop at the next breakpoint.
Set Start Point
If you have a really long test and it’s failing towards the end, then you can set a custom start point so that you don’t have to run the entire test when you’re investigating the failure. For example, your test might register a new user, log in, and then fail on the welcome page. You could simply navigate to the welcome page yourself and set your test to start from there. To set a start point simply right click on the first command you want Selenium IDE to execute and click ‘Set / Clear Start Point’. You will see a small play icon appear to the left of your command.
Pause on Failure
If you like the idea of Selenium IDE pausing whenever it encounters a failure, then I recommend Samit Badle’s plugin Power Debugger, which adds an icon to the toolbar. When enabled, Selenium IDE will pause on each failure allowing you to investigate the cause.
It’s not uncommon that when in ‘debug mode’ tests actually start passing. This is often due to the speed at which tests are executed. Stepping through manually allows extra time for each command to finish executing. One I’ve seen all too often is a
click that causes a new page to load followed immediately by something like
assertTitle for the new page. When run fast there’s no time for the page to load, but when executed slowly the test might pass. Look out for issues that are related to execution speed.
In conclusion, there are many ways that Selenium IDE can assist you in investigating failures and debugging your tests. There are even more powerful ways to do this when your tests mature out of Selenium IDE and into a programming or scripting language. For example, many test frameworks create useful reports that replace the log in Selenium IDE, and give far more detail regarding the failures. You can also do more exciting things like take screenshots or videos of your tests running!
If you have any questions you might find the Selenium Users group useful.
If you find any issues in Selenium please check if it’s already been raised on the official issue tracker before raising a new report with as much information as possible.