Stuart Thomas
Stuart Thomas
Engineering Leadership Consultant
He / Him
Duck enthusiast, technology leader, and quality advocate focused on building high-autonomy teams and sustainable delivery.
🎂 MoTaBirthday | December 20, 2022
Open To
Speak
Write
Teach
Mentor
CV Reviews
Today I processed a large chunk of all the things from MoTaCon. Post-MoTaCon often feels like the start of a new year. A chance to catch our breath and to process all the things. And then start the...
10 Oct
After a while overthinking about what I was going to write to summarise my first MotaCon experience, I decided to just do it! No overthinking*, no AI**, no plan - Very much like my cringe 99 second...
7 Oct
Inspired by Dan Ashby's Continuous Testing model, and Janet Gregory and Lisa Crispin's Holistic Testing Model, Stuart demonstrates how the 5 Shifts of AI fit in. 1. Input shift - Synthetic Per...
6 Oct
The 5 Shifts for Testing AI1. The Input Shift2. The Verification Shift3. The Safety Shift4. The Baseline Shift5. The Observability ShiftThanks for such an insightful talk Stuart!
6 Oct
Integrate monitoring, observability, and alerting into the core quality engineering process to ensure systems are as diagnosable as they are functional
21 Apr
63% of you were not aware of the existence of the Staff Quality Engineer role
24 Apr
Understand the strategic concepts required to tame the chaos of testing AI by understanding the behaviour of the beast.
9 Oct
Transform your team’s delivery by learning how small, consistent improvements in testing, automation, and collaboration can make everything better for both your team and users
20 Oct
Synthetic Personas
"First we borrow the idea of personas from exploratory testing. We build a suite of different personas that represent different elements of our user base. We use those personas to build automated persona driven testing. This takes our well-defined personas and uses them to guide an LLM to act like those users. On top of that, you feed the persona a goal and set it run against your chat interface until it achieves its goal, or x turns have completed."So what? A few hand-written prompts only cover what the author thought of; synthetic personas generate conversations closer to the creative, messy ways real users behave, broadening coverage of an AI product. Synthetic inputs are a supplement to real ones, not a replacement.Examples: Defining an impatient persona who gives only half the information and switches topic partway through; running one persona with the goal of booking a refund until it succeeds or hits ten turns.
Golden Dataset
"A curated set of inputs, each with the expected behaviour or reference answer that your team has agreed represents good. You run the system against it, score the results (with code, judges, humans, or a mix) and get numbers you can compare release over release."So what? When you cannot define a single correct response, a golden dataset gives a stable baseline to tell whether a change (a new model, an edited prompt, altered retrieval) improved things or made them worse. Examples: Building a few hundred cases from production traces, support tickets and expert-authored hard cases, tagged by category so results can be sliced; holding back a portion of cases so prompt tuning does not overfit to the whole set.
Governance
The policies, processes, and guardrails put in place to ensure that code, experiments, and tools produced across a team meet standards of safety, consistency, and reliability, especially as AI lowers the barrier to building.
The question is no longer about engineering capacity, but Governance. The teams that win will be those that create the safest “Playgrounds”. The teams that win won’t write the most code — they’ll create the safest environments for everyone to build in.
Ephemeral Code
Short-lived, AI-generated code designed to be deleted after use, typically for running experiments such as A/B tests on a component or call-to-action, with Engineering providing guardrails through a component library and code review.
Disposable Code
AI-generated code used for internal validation and testing core logic during the "can we even do this?" phase. It lives in a sandbox and never reaches production; if it works, the learnings are handed to Engineering to be rebuilt properly.
Comments
(1)
Head of Engineering Lesson 1 - Communicating with the Exec team
This resonates so much with me currently, both in the work I am doing and something I am writing. In leadership there is a need to be able to "tell stories" to help others understand, but also those stories have to be tailored to the audience, and the exec definitely tend to want and need to TL;DR version!
Testing the Untestable: Building a strategy for testing AI
(www.thequalityduck.co.uk)
07 Oct
Understand exploratory testing, and unlock its true value
(www.thequalityduck.co.uk)
28 Jul
Not registered for any upcoming events
Thanks for your contribution
Testing the untestable: building a strategy for testing AI
Community Star
about 1 hour ago
Learner completed your lesson
3.2.0 of MoT Software Quality Engineering Certificate
Community Star
1 day ago
Sticking the MoTaCon clipboards
Today I processed a large chunk of all the things from MoTaCon. Post-MoTaCon often feels like the start of a new year. A chance to catch our breath and to process all the things. And then start the...
Moment
1 day ago
