Today I processed a large chunk of all the things from MoTaCon. Post-MoTaCon often feels like the start of a new year. A chance to catch our breath and to process all the things. And then start the...

"First we borrow the idea of personas from exploratory testing. We build a suite of different personas that represent different elements of our user base. We use those personas to build automated persona driven testing. This takes our well-defined personas and uses them to guide an LLM to act like those users. On top of that, you feed the persona a goal and set it run against your chat interface until it achieves its goal, or x turns have completed."So what? A few hand-written prompts only cover what the author thought of; synthetic personas generate conversations closer to the creative, messy ways real users behave, broadening coverage of an AI product. Synthetic inputs are a supplement to real ones, not a replacement.Examples: Defining an impatient persona who gives only half the information and switches topic partway through; running one persona with the goal of booking a refund until it succeeds or hits ten turns.

"A curated set of inputs, each with the expected behaviour or reference answer that your team has agreed represents good. You run the system against it, score the results (with code, judges, humans, or a mix) and get numbers you can compare release over release."So what? When you cannot define a single correct response, a golden dataset gives a stable baseline to tell whether a change (a new model, an edited prompt, altered retrieval) improved things or made them worse. Examples: Building a few hundred cases from production traces, support tickets and expert-authored hard cases, tagged by category so results can be sliced; holding back a portion of cases so prompt tuning does not overfit to the whole set.

Understand the strategic concepts required to tame the chaos of testing AI by understanding the behaviour of the beast.

After a while overthinking about what I was going to write to summarise my first MotaCon experience, I decided to just do it! No overthinking*, no AI**, no plan - Very much like my cringe 99 second...

Inspired by Dan Ashby's Continuous Testing model, and Janet Gregory and Lisa Crispin's Holistic Testing Model, Stuart demonstrates how the 5 Shifts of AI fit in. 1. Input shift - Synthetic Per...

The 5 Shifts for Testing AI1. The Input Shift2. The Verification Shift3. The Safety Shift4. The Baseline Shift5. The Observability ShiftThanks for such an insightful talk Stuart!

Stuart’s talk inspired many ideas for how we can improve how we test AI and make the untestable testable! Thank you 🙏 

An insightful talk from Stuart Thomas on how to create a testing strategy for AI. In this slide, he gave an overview of the five shifts in AI.

Great to see Stuart Thomas on stage.So many interesting thoughts and ideas coming from his talk.Centre to his talk he is explaining the “5 shifts of testing ai”.

As part of The Input Shift, Stuart Thomas talks about automated persona testing, and how it can handle many things at scale.It brings different challenges, such as the the need to verify huge amoun...