AI-powered testing: supercharging the human tester

Caitlin has spent the past year putting Playwright’s AI agents through their paces, and this session is her honest field report. She covers the three agents Playwright currently offers: the planner, which writes spec or BDD-style test plans; the generator, which turns them into code; and the healer, which tries to repair tests when they fail.

To test the planner, Caitlin pointed it at the data tables on a well-known locator practice site and compared three leading models against a golden test set she built herself. The results were revealing. Claude hit the right total but missed key coverage. GPT over-planned, which Caitlin prefers, since filtering extra tests beats not knowing what’s missing. Gemini fell short. Two models also added redundant selector tests and, without being asked, accessibility tests, despite nothing in the prompt, site or DOM pointing that way. 

Her verdict: the planner cuts the slog of writing specs by more than half, but a human still needs to review what it produces.

She rates the generator highly, provided every generated test gets human review on top of any agent checks. The healer is her favourite and the most fragile of the three. It hunts the DOM for broken selectors, but it assumes the test logic is correct, struggles with complex business requirements, and falls apart on test libraries built over component libraries, where it bloats suites with unreusable code. Once a healed test passes, can you trust it?

The Q&A went further. Caitlin advised a team moving from back-end testing into Playwright to bring in an automation tester before diving in, and made an enthusiastic case for Playwright’s API testing over Postman. Asked whether self-healing tests could hide communication breakdowns in a team, she agreed they can, but noted that healing runs are logged, so recurring patterns can still be spotted and raised. Asked whether AI helps or hinders testing, she said it depends on who’s using it, and on whether the business sees AI as support for testers or as a way to cut headcount.

The takeaway: AI speeds up the dull parts of testing, but human judgement is what makes the output worth trusting.