A reference point used to decide whether a test has passed or failed. For LLM testing this becomes unreliable because multiple valid outputs can exist for the same input. So what? The absence of a stable oracle is one of the central challenges of AI testing. Techniques like metamorphic testing exist partly to work around it by checking consistency rather than correctness. Example: "Who was the first president of the USA?" has a clear oracle. A summarisation request does not. "The 'expected result' can be determined, but will always have some ambiguity. Comparing it to the 'actual result' won't be as straightforward as you are used to." — Demi Van Malcot

A machine learning technique that uses layered neural networks to find patterns across large volumes of data. LLMs are built on deep learning to make connections across billions of words and generate contextually relevant responses. So what? LLMs work by statistical pattern-matching rather than reasoning — a foundational insight for anyone designing tests. Example: An LLM predicts the most statistically likely next word or phrase, not the most factually accurate one. "They are trained on billions of words from different sources. Using deep learning, they make connections between all the words they are trained on to answer whatever questions we ask of them." — Demi Van Malcot

A category of AI model that produces new content: text, images, code, or other outputs, in response to input prompts, rather than returning a fixed or pre-programmed answer. LLMs are the most widely used type of generative AI. So what? Because outputs are generated fresh each time, the testing approaches used for traditional deterministic software don't transfer cleanly. Concepts like "expected result" and pass/fail need to be rethought. Example: ChatGPT, Claude, Bard, and Copilot are all generative AI applications built on large language models. "It won't give an answer based on what is logically correct, but on what is statistically most likely. The sentences can be completely correct, while the answer is completely wrong." — Demi Van Malcot

Learn how to evaluate LLM quality and limitations using a range of testing techniques, from unit and regression testing to bias, adversarial and explainability testing.

We have recreated the import screens users need to fill in for declaration in our test environment. Although I appreciate the devs wanting to make it as close to real as possible, I have to check t...

In TWiQ today, Aj raised the idea of self-service infrastructure and it got me thinking about how that is related to platform engineering. Something I realise I don't know enough about, but a recen...

Inspired by my manager who explained I 'monkey-barred' to a new job (before I was a tester I was a baker) and couldn't really transfer a lot of skills with me. I looked at him baffled because not o...

TWiQ episodes now sounds more cool when we see our hosts with cameras on and their entry reminds me of animations - Flying In from Left and Right :) Cool discussions about negative tests, happ...

Live experimentation, technical glitches, and community challenges along the way

Today's This Week in Quality had the hosts with their cameras on for the very first time! ... And since I could see them, I was extremely convinced that they could see me as well. Cue lots of emoti...

A This Week in Quality (TWiQ) evolution, we switched on the cameras for the first time! Thank you to Demi, Preeti and Gary for being the first people to do that. 🎉Do listen and now watch the episod...

This Week in Quality enters a new, cameras-on format