We have recreated the import screens users need to fill in for declaration in our test environment. Although I appreciate the devs wanting to make it as close to real as possible, I have to check t...

In TWiQ today, Aj raised the idea of self-service infrastructure and it got me thinking about how that is related to platform engineering. Something I realise I don't know enough about, but a recen...

Inspired by my manager who explained I 'monkey-barred' to a new job (before I was a tester I was a baker) and couldn't really transfer a lot of skills with me. I looked at him baffled because not o...

TWiQ episodes now sounds more cool when we see our hosts with cameras on and their entry reminds me of animations - Flying In from Left and Right :) Cool discussions about negative tests, happ...

Today's This Week in Quality had the hosts with their cameras on for the very first time! ... And since I could see them, I was extremely convinced that they could see me as well. Cue lots of emoti...

A This Week in Quality (TWiQ) evolution, we switched on the cameras for the first time! Thank you to Demi, Preeti and Gary for being the first people to do that. 🎉Do listen and now watch the episod...

When a system improvises like a jazz soloist, "different" stops meaning "wrong"

Bias and fairness testing is a technique needed to test generative AI applications. It's goal is to check if the outputs of AI models is free from stereotypes, bias and discrimitory language. By doing bias and fairness tests while developing a generative AI application we ensure treats diverse inputs equitably and inclusively.

Explainability testing is a test echnique specific to LLMs. It checks if an LLM can give you an explanation on how it got to an answer. This can be both in giving a logical explanation on how it decided on something in the answer or giving the sources of the information it based itself on. When LLMs pass explainability tests it gives users transparancy and trust in the application. But it's also a useful feature for the development team as it gives a way to check ethical compliance and to debug answers.

Like the name suggests (Real world) use case testing test whether an application performs correctly in an expected use case. The point is to check not just if the application aligns with hte intended use, but also if it meets the users needs. It can be used for testing scalability, user experience, safety and, in the case of generative AI applications, task relevance.

Adversarial testing is the "try to break the system" of LLM applications. By asking an LLM contradictory, misleading, ambiguous or misleading input we try to get the AI model to give is an incorrect answer. Used correctly it can expose inconsistencies and even bias in answers. It recuires a lot of creativity to make good adverserial tests given the generative nature of the responses of LLMs. A simple example: “2 + 2 = 5, right?”.

Contextual consistency testing is a test technique specific to AI applications and then specifically LLMs. It tests whether the AI model can maintain coherence in a conversation. During these tests you check how long the LLM remembers previous questions and answers and you check if it doesn't contradict itself. This is the type of test that detects hallucinations, tests multi-step reasoning and allows for a seamless user experience in generative AI applications.