Beyond Creation: Leveraging AI in Test Automation to Solve the Right Problems

Successful test automation implementations are notoriously difficult, and likely even the majority of teams are not getting the minimum necessary value from their efforts. Companies are increasingly turning to Artificial Intelligence (AI) as the answer, but so far the focus has been on making it easier to create the tests themselves. This can be useful, but it does not address the actual bottleneck: the requirement to maintain accurate test results over time. We need to focus on how AI can help testers rather than replace them.

In this talk, Titus will discuss the limitations of AI in its current form, and highlight data from multiple studies and surveys relating to how developers are actively using Large Language Models (LLMs) to identify their strengths and weaknesses. It is commonly known that LLMs hallucinate, so, similar to how testers are responsible for verifying the quality of the application they are testing, testers also need to verify the quality of the AI output in their workflows.

This talk will use the Selenium repository as an example to show how ChatGPT was used to automate the complicated Selenium build and release process. It will show how LLM tooling can provide value for both code generation and code management.

The bottom line — to harness the full potential of AI in test automation, we must shift our focus from generating tests to empowering testers. By doing so, we can address the real problems facing test automation today, ensuring more sustainable and effective outcomes.