Demi Van Malcot
Demi Van Malcot
Test engineer, Test lead, Quality manager
she/her
I've been in testing since 2023, since then I never stopped learning and taking every opportunity I've come across. From becoming test lead not long after I started, to being a community lead for testing and for AI in at the company I work at. Nowadays I'm learning the ropes of leading with quality as I have added the role of quality manager of my department to my growing list of titles.
π MoTaBirthday | October 25, 2024
Open To
Speak
Write
Podcasting
An evening filled with conversations, discussions, exchanging thoughts with known and new faces. The video is just a snapshot - in reality it was much more fun filled, energetic and loads of ...
11 Oct
Harsh Gaudana ran a "Unlocking the World through Inclusive Design" workshop in the main hall MoTaCon social hall:
Welcome to AID-X Lab (Accessibility & Inclusive Design Experience Lab), a fast-...
5 Oct
If anyone had any doubts what a MoTaverse karaoke sounds like in a supportive community such as ours...heres a clip. What an amazing evening. Music is very special to me and just being with these g...
2 Oct
For all the people who missed out on the karaoke: here are some pictures of how much fun it was
2 Oct
Learn how to evaluate LLM quality and limitations using a range of testing techniques, from unit and regression testing to bias, adversarial and explainability testing.
23 Jun
When a system improvises like a jazz soloist, "different" stops meaning "wrong"
3 Jun
A warm pre-MoTaCon episode full of conference anticipation, community stories, practical advice for first-time attendees, and excitement around stars, selfies, Moments, meetups, and merch.
25 Sep
A lively return episode catching up on MoTaCon preparations, community updates, AI overwhelm, demo-day disasters, knowledge sharing, and why human connection remains central to quality work.
18 Sep
Non-deterministic Output
Behaviour where the same input can produce different outputs across multiple runs. LLMs exhibit this by design, requiring testers to shift from exact-match checks to presence/absence assertions. So what? Non-determinism means a regression test can "fail" without the answer being wrong β a fundamental difference from testing traditional software. Example: Running the same summarisation prompt twice may return the same meaning in different words. Editorial note: This definition has been inferred from how the term was used in the source material.
"The maximum time you expect your application to take to respond to a specific request or action." β Demi Van Malcot
Test Oracle
A reference point used to decide whether a test has passed or failed. For LLM testing this becomes unreliable because multiple valid outputs can exist for the same input. So what? The absence of a stable oracle is one of the central challenges of AI testing. Techniques like metamorphic testing exist partly to work around it by checking consistency rather than correctness. Example: "Who was the first president of the USA?" has a clear oracle. A summarisation request does not. "The 'expected result' can be determined, but will always have some ambiguity. Comparing it to the 'actual result' won't be as straightforward as you are used to." β Demi Van Malcot
Deep Learning
A machine learning technique that uses layered neural networks to find patterns across large volumes of data. LLMs are built on deep learning to make connections across billions of words and generate contextually relevant responses. So what? LLMs work by statistical pattern-matching rather than reasoning β a foundational insight for anyone designing tests. Example: An LLM predicts the most statistically likely next word or phrase, not the most factually accurate one.
"They are trained on billions of words from different sources. Using deep learning, they make connections between all the words they are trained on to answer whatever questions we ask of them." β Demi Van Malcot
Generative AI
A category of AI model that produces new content: text, images, code, or other outputs, in response to input prompts, rather than returning a fixed or pre-programmed answer. LLMs are the most widely used type of generative AI. So what? Because outputs are generated fresh each time, the testing approaches used for traditional deterministic software don't transfer cleanly. Concepts like "expected result" and pass/fail need to be rethought. Example: ChatGPT, Claude, Bard, and Copilot are all generative AI applications built on large language models.
"It won't give an answer based on what is logically correct, but on what is statistically most likely. The sentences can be completely correct, while the answer is completely wrong." β Demi Van Malcot
Bias and fairness testing
Bias and fairness testing is a technique needed to test generative AI applications. It's goal is to check if the outputs of AI models is free from stereotypes, bias and discrimitory language. By doing bias and fairness tests while developing a generative AI application we ensure treats diverse inputs equitably and inclusively.
Comments
(13)
Ady asks the MoTaverse #4 Mindset chapter draft
I found it very useful and interesting, I especially like how clear the examples and excersizes are.
At some points/sentences I am unsure on whether it is a (sub)title and the formatting didn't copy or if you meant it to be jsut a sentence (in that case I do think it's missing subtitles).
What I miss is some variety for some fo the examples. In the paragraph that begins with "You have probably heard of some of the examples..." many exampels seem very similar in what wasn't thought about in that example. I think it needs more variety (or just more examples?).
I'm also missing some more concrete steps on how to start getting this mindset. Was expecting that in "how do I get better", but what follows isn't about that. Is it in a next chapter maybe?
Also wondering if the person indicated in "what works for me" and "how do I get better" are the same persons.
I am interested in reading more!
Ady asks the MoTaverse #3
This graphic is more modern and simplistic than the other one (I liek the colors better).
But it's somehow giving a different meaning to the model π€The other graphic made it seem liek just a lsit of the mindsets and what's included there. This one makes me think that the mindsets are connected and as a consequences there are some middle grounds between the mindsets that ar enext to each other. Also makign the diagonal pairs seem like opposites of each other.
Not sure if that is what you're going for.
Want to practise API testing? Try an API that lies to you
As someone who is still hesitant about APIs (they are weird creatures I still don't fully comprehend π) this sounds frightening and intriguing at the same time
Ady asks the MoTaverse #2
For Dark and Aggressive I would put more then just Malevolant. For example glass half empty or doomsday thinking also counts as dark, but isn't really malevolant.
Also is Empirical included in Scientific mindset? And intuitive in Exploratory?
Ady asks the MoTaverse #1
I agree with Gary and Brandy, the first one immediately grabbed my attention and made me think it would be an interesting book.
Not registered for any upcoming events
Thanks for your contribution
An Evening of Connections - Dinner Organised by TestMu AI
Community Star
about 21 hours ago
Thanks for your contribution
An Evening of Connections - Dinner Organised by TestMu AI
Community Star
1 day ago
Leading with Quality 2026 Attendee
This badge is awarded to those who have attended Leading with Quality 2026.
Badge Award
1 day ago
MoTaCon 2026 Attendee
This badge is awarded to those who have attended MoTaCon 2026.
Badge Award
1 day ago
