Demi Van Malcot
Demi Van Malcot
Test engineer, Test lead, Quality manager
she/her
I've been in testing since 2023, since then I never stopped learning and taking every opportunity I've come across. From becoming test lead not long after I started, to being a community lead for testing and for AI in at the company I work at. Nowadays I'm learning the ropes of leading with quality as I have added the role of quality manager of my department to my growing list of titles.
🎂 MoTaBirthday | October 25, 2024
Open To
Speak
Write
Podcasting
At TWiQ, I mentioned how I managed to get multiple teams doing demonstrations for each other to make work more visible. One team found that another team had solved their problem and learned from th...
23 Oct
Ady Stokes is dogsitting this week on the paw-rific Dexter. The canine joined us live on stage during This Week in Quality!
23 Oct
Had to have a play based on something said in TWiQ today on AI introducing the hosts. So here it is 😂Shawn Vernier"Today we're joined by Shawn Vernier, a Quality Engineer, community leader, and pas...
23 Oct
I went to a one-day conference last week (BrewtCon) and brought back a bunch of fun stickers.All the extras were brought to the office for anyone who wanted them.One of them read 'Silently judging ...
20 Oct
Learn how to evaluate LLM quality and limitations using a range of testing techniques, from unit and regression testing to bias, adversarial and explainability testing.
23 Jun
When a system improvises like a jazz soloist, "different" stops meaning "wrong"
3 Jun
The gang explore security vulnerabilities, the challenges of testing randomness and AI, and how better communication and collaboration can strengthen quality across teams.
23 Oct
A warm pre-MoTaCon episode full of conference anticipation, community stories, practical advice for first-time attendees, and excitement around stars, selfies, Moments, meetups, and merch.
25 Sep
Non-deterministic Output
Behaviour where the same input can produce different outputs across multiple runs. LLMs exhibit this by design, requiring testers to shift from exact-match checks to presence/absence assertions. So what? Non-determinism means a regression test can "fail" without the answer being wrong — a fundamental difference from testing traditional software. Example: Running the same summarisation prompt twice may return the same meaning in different words. Editorial note: This definition has been inferred from how the term was used in the source material.
"The maximum time you expect your application to take to respond to a specific request or action." — Demi Van Malcot
Test Oracle
A reference point used to decide whether a test has passed or failed. For LLM testing this becomes unreliable because multiple valid outputs can exist for the same input. So what? The absence of a stable oracle is one of the central challenges of AI testing. Techniques like metamorphic testing exist partly to work around it by checking consistency rather than correctness. Example: "Who was the first president of the USA?" has a clear oracle. A summarisation request does not. "The 'expected result' can be determined, but will always have some ambiguity. Comparing it to the 'actual result' won't be as straightforward as you are used to." — Demi Van Malcot
Deep Learning
A machine learning technique that uses layered neural networks to find patterns across large volumes of data. LLMs are built on deep learning to make connections across billions of words and generate contextually relevant responses. So what? LLMs work by statistical pattern-matching rather than reasoning — a foundational insight for anyone designing tests. Example: An LLM predicts the most statistically likely next word or phrase, not the most factually accurate one.
"They are trained on billions of words from different sources. Using deep learning, they make connections between all the words they are trained on to answer whatever questions we ask of them." — Demi Van Malcot
Generative AI
A category of AI model that produces new content: text, images, code, or other outputs, in response to input prompts, rather than returning a fixed or pre-programmed answer. LLMs are the most widely used type of generative AI. So what? Because outputs are generated fresh each time, the testing approaches used for traditional deterministic software don't transfer cleanly. Concepts like "expected result" and pass/fail need to be rethought. Example: ChatGPT, Claude, Bard, and Copilot are all generative AI applications built on large language models.
"It won't give an answer based on what is logically correct, but on what is statistically most likely. The sentences can be completely correct, while the answer is completely wrong." — Demi Van Malcot
Bias and fairness testing
Bias and fairness testing is a technique needed to test generative AI applications. It's goal is to check if the outputs of AI models is free from stereotypes, bias and discrimitory language. By doing bias and fairness tests while developing a generative AI application we ensure treats diverse inputs equitably and inclusively.
Comments
(16)
The MoTaverse helps you with squish that imposter syndrome away
I often feel like I'm just doing 'something', what comes to mind, improvising. This makes me very insecure as I don't feel like I know what I'm doing. However all these instinctual things I did, are all actual things. MoT has helped me find the word for it and to improve on it. If there wasn't a word for it, the community would find out e for it, thus making it feel more official.
Also when I tried to achieve something g the experiences of other people in the community helped me realise I'm not alone. Often people shared advice on how to deal with it or how to find the problem.
All in all MoT helped me find the toolset I needed (or maybe is the toolset 🤔) to find the confidence, agency and vocabulary I needed to remedy imposter syndrome.
Ady asks #6 Updated Mindsets visual heuristic
Love the connection between neighbouring mindsets!
It does make me wonder if that means you consider diagonal mindsets as opposites (so analytical and grounded is the opposite of visionary and open)? Is there a way they still overlap? Is the purple circle in the middle jsut the title or whete things meet in a venn diagram?
Ady asks #5 Mindset chapter draft updated after feedback
I like allt he improvements you made. Already lookign forward to the rest of the book 😉
Ady asks the MoTaverse #4 Mindset chapter draft
I found it very useful and interesting, I especially like how clear the examples and excersizes are.
At some points/sentences I am unsure on whether it is a (sub)title and the formatting didn't copy or if you meant it to be jsut a sentence (in that case I do think it's missing subtitles).
What I miss is some variety for some fo the examples. In the paragraph that begins with "You have probably heard of some of the examples..." many exampels seem very similar in what wasn't thought about in that example. I think it needs more variety (or just more examples?).
I'm also missing some more concrete steps on how to start getting this mindset. Was expecting that in "how do I get better", but what follows isn't about that. Is it in a next chapter maybe?
Also wondering if the person indicated in "what works for me" and "how do I get better" are the same persons.
I am interested in reading more!
Ady asks the MoTaverse #3
This graphic is more modern and simplistic than the other one (I liek the colors better).
But it's somehow giving a different meaning to the model 🤔The other graphic made it seem liek just a lsit of the mindsets and what's included there. This one makes me think that the mindsets are connected and as a consequences there are some middle grounds between the mindsets that ar enext to each other. Also makign the diagonal pairs seem like opposites of each other.
Not sure if that is what you're going for.
TWiQ or Treat
Share your week’s ghostly highlights, ghoulish challenges, and spooky lessons in quality
This Week in Quality
about 6 hours ago
The MoTaverse helps you with squish that imposter syndrome away
"I often feel like I'm just doing 'something', what comes to mind, improvising. This makes me very insecure as I don't feel like I know what I'm doing. However all these instinctual things I did, are all actual things. MoT has helped me find the word fo..."
Moment
1 day ago
