Roundtable: AI Learnings and Demos

In this AI sharing, learning and demo session we work through where AI has been earning its place in a testing workflow. Nataliia described her push to keep AI activity embedded in the same repository as the code it supports, and walked through an in-progress multi-agent workflow for backfilling test coverage on a high-risk access control project, plus a PR risk-flagging experiment that caught a regression the team had previously caught too late. Her stated preference throughout was for AI as an analysis and pattern-spotting tool rather than a generator of test cases, which she sees as a job better suited to humans on her team right now.

The conversation's throughline, raised by Stephen early on, was the unresolved question of how anyone actually knows whether an AI agent is doing a good job without predefined success criteria. That surfaced repeatedly: the gap between a well-structured test case and a useful one, the risk of AI-generated volume overwhelming the human reviewer, a proposed experiment to test agents against deliberately planted defects, and a closing analogy comparing AI hype to the supplement industry (real benefits exist, but unproven without personal measurement). 

Jonathan Cole also joined in as they also worked through practical ground, like what actually distinguishes an "agent" from a custom prompt or Claude Chat project, and closed on the idea that AI is best used for things there simply isn't time to do otherwise, such as background exploratory testing or baseline security checks, rather than replacing work that's already going fine.

Session Chapters:

1. 0:00 Welcome and session framing 
2. 5:30 Bringing AI output into the daily workflow
3. 12:00 A three agent workflow for backfilling tests 
4. 19:00 Catching regressions with PR risk flagging 
5. 24:00 How do you know it's doing a good job
6. 31:00 Business value, KPIs and a planted defect test 
7. 39:00 Requirements, doom loops and model comparisons 
8. 46:00 Taming AI's output volume 
9. 53:00 Defining agents and where to prioritize real use cases