Behaviour where the same input can produce different outputs across multiple runs. LLMs exhibit this by design, requiring testers to shift from exact-match checks to presence/absence assertions. So what? Non-determinism means a regression test can "fail" without the answer being wrong — a fundamental difference from testing traditional software. Example: Running the same summarisation prompt twice may return the same meaning in different words. Editorial note: This definition has been inferred from how the term was used in the source material. "The maximum time you expect your application to take to respond to a specific request or action." — Demi Van Malcot

A reference point used to decide whether a test has passed or failed. For LLM testing this becomes unreliable because multiple valid outputs can exist for the same input. So what? The absence of a stable oracle is one of the central challenges of AI testing. Techniques like metamorphic testing exist partly to work around it by checking consistency rather than correctness. Example: "Who was the first president of the USA?" has a clear oracle. A summarisation request does not. "The 'expected result' can be determined, but will always have some ambiguity. Comparing it to the 'actual result' won't be as straightforward as you are used to." — Demi Van Malcot

A machine learning technique that uses layered neural networks to find patterns across large volumes of data. LLMs are built on deep learning to make connections across billions of words and generate contextually relevant responses. So what? LLMs work by statistical pattern-matching rather than reasoning — a foundational insight for anyone designing tests. Example: An LLM predicts the most statistically likely next word or phrase, not the most factually accurate one. "They are trained on billions of words from different sources. Using deep learning, they make connections between all the words they are trained on to answer whatever questions we ask of them." — Demi Van Malcot

A category of AI model that produces new content: text, images, code, or other outputs, in response to input prompts, rather than returning a fixed or pre-programmed answer. LLMs are the most widely used type of generative AI. So what? Because outputs are generated fresh each time, the testing approaches used for traditional deterministic software don't transfer cleanly. Concepts like "expected result" and pass/fail need to be rethought. Example: ChatGPT, Claude, Bard, and Copilot are all generative AI applications built on large language models. "It won't give an answer based on what is logically correct, but on what is statistically most likely. The sentences can be completely correct, while the answer is completely wrong." — Demi Van Malcot

Bias and fairness testing is a technique needed to test generative AI applications. It's goal is to check if the outputs of AI models is free from stereotypes, bias and discrimitory language. By doing bias and fairness tests while developing a generative AI application we ensure treats diverse inputs equitably and inclusively.

Explainability testing is a test echnique specific to LLMs. It checks if an LLM can give you an explanation on how it got to an answer. This can be both in giving a logical explanation on how it decided on something in the answer or giving the sources of the information it based itself on. When LLMs pass explainability tests it gives users transparancy and trust in the application. But it's also a useful feature for the development team as it gives a way to check ethical compliance and to debug answers.

Like the name suggests (Real world) use case testing test whether an application performs correctly in an expected use case. The point is to check not just if the application aligns with hte intended use, but also if it meets the users needs. It can be used for testing scalability, user experience, safety and, in the case of generative AI applications, task relevance.

Adversarial testing is the "try to break the system" of LLM applications. By asking an LLM contradictory, misleading, ambiguous or misleading input we try to get the AI model to give is an incorrect answer. Used correctly it can expose inconsistencies and even bias in answers. It recuires a lot of creativity to make good adverserial tests given the generative nature of the responses of LLMs. A simple example: “2 + 2 = 5, right?”.

Contextual consistency testing is a test technique specific to AI applications and then specifically LLMs. It tests whether the AI model can maintain coherence in a conversation. During these tests you check how long the LLM remembers previous questions and answers and you check if it doesn't contradict itself. This is the type of test that detects hallucinations, tests multi-step reasoning and allows for a seamless user experience in generative AI applications.

Behavioral testing checks if an application behaves as intended in realistic situations. It resembles real-world use case testing in that we check how users will use the application. The difference being that behavioral testing tests a specific functionality while real-world use case testing tests an entire workflow. It resembles unit testing as both test a granluar part of the application. The difference being the perspective, unit testing is from the viewpoint of the code, where behavioral testing is from the viewpoint of the user.It's a test technique that lends itself very well for testing generative AI applications, for example how well the instructions in the input have been followed to create the output. 

Stress testing is a specific kind of performance testing. Where performance testing works on speed, responsiveness and reliability in general (although usually under normal circumstances), stress testing is trying to find the breaking point. How does the system behave in peak moments, how does it handle large amounts of data or big files, does it 'fail gracefully'? In stress testing we try to determine the limit of the application, the areas that might break under heavy stress and how well a system recovers after failure. 

Code that was written before current practices, tooling, or team knowledge were in place, and that typically lacks unit tests, up-to-date documentation, or clear structure. Legacy code is not necessarily broken; it may run perfectly well, but its internals are often difficult to understand or safely change. Demi described a migration project involving an old application with no tests, outdated or multilingual documentation, and logic nobody fully understood. The challenge with legacy code is deciding how much to clean before making it work, when deadlines are real and touching it carries risk.