Learnings from TestBash: Using AI in Test Automation to Achieve Continuous Testing by Titus Fortner

On the slide, Titus shows the audience the trends that he has identified in the testing industry. This photo was taken when he asked the audience; “how many people here have a role where their primary responsibility is exploratory testing?” and “how many people have a role where their primary responsibility is writing code and automated testing?” Either way, Titus argues manual / exploratory testing doesn't scale and that we can use automation and AI to solve the right problems.

For exploratory testing, AI can have large positive / helpful impacts on test case creation but it isn't a magic solution that replaces shift-left activities like requirement analysis.

For automation, Titus explains how automated tests come with costs (creation, maintenance and execution). Titus talks about test maintenance and the types of bugs that automation can find:
- Type I errors are where the feature works as expected, but the test has an issue.
- Type II errors are where the feature does not work as expected, but the test does not detect the issue.
- Real regression bugs are found when the feature does not work as expected, and the test fails.

Used incorrectly, AI can hide "type I errors" or exaggerate "type II errors" through hallucinations, so he emphasizes critical thinking around AI tools and the importance of using AI to solve the correct problems for your team and context.