Demi Van Malcot
Demi Van Malcot
Test engineer, Test lead, Quality manager
she/her
I've been in testing since 2023, since then I never stopped learning and taking every opportunity I've come across. From becoming test lead not long after I started, to being a community lead for testing and for AI in at the company I work at. Nowadays I'm learning the ropes of leading with quality as I have added the role of quality manager of my department to my growing list of titles.
π MoTaBirthday | October 25, 2024
Open To
Speak
Write
Meet at MoTaCon 2026
Podcasting
1,000 songs in your pocket changed how the world listened to music. The MoTaverse changes how tech professionals grow their careers.Sixteen features, sixteen stories, one thread running through al...
26 Aug
Sneaky bug sneaking on the TWiQ stage!
7 Aug
I absolutely love all the virtual events. There is a variety of events that have different levels of participation. Whether you just want to listen, chat with everyone or are ready to join the sta...
3 Aug
I'm the kind of tester who is curious about any and every button. What do they do,? What happens when I press it? What happens when I press all of them at the same time? Why is it here?Sometimes cr...
17 Jul
Learn how to evaluate LLM quality and limitations using a range of testing techniques, from unit and regression testing to bias, adversarial and explainability testing.
23 Jun
When a system improvises like a jazz soloist, "different" stops meaning "wrong"
3 Jun
A lively episode about MoTaverse profile upgrades, MoTaCon momentum, AI-assisted testing, hidden knowledge silos, and why real human conversations still matter when tools move fast.
7 Aug
A 'good problems' episode about production-testing drama, MoTaverse growth, new voices, lifelong learning, and how community conversations help people keep going.
24 Jul
Non-deterministic Output
Behaviour where the same input can produce different outputs across multiple runs. LLMs exhibit this by design, requiring testers to shift from exact-match checks to presence/absence assertions. So what? Non-determinism means a regression test can "fail" without the answer being wrong β a fundamental difference from testing traditional software. Example: Running the same summarisation prompt twice may return the same meaning in different words. Editorial note: This definition has been inferred from how the term was used in the source material.
"The maximum time you expect your application to take to respond to a specific request or action." β Demi Van Malcot
Test Oracle
A reference point used to decide whether a test has passed or failed. For LLM testing this becomes unreliable because multiple valid outputs can exist for the same input. So what? The absence of a stable oracle is one of the central challenges of AI testing. Techniques like metamorphic testing exist partly to work around it by checking consistency rather than correctness. Example: "Who was the first president of the USA?" has a clear oracle. A summarisation request does not. "The 'expected result' can be determined, but will always have some ambiguity. Comparing it to the 'actual result' won't be as straightforward as you are used to." β Demi Van Malcot
Deep Learning
A machine learning technique that uses layered neural networks to find patterns across large volumes of data. LLMs are built on deep learning to make connections across billions of words and generate contextually relevant responses. So what? LLMs work by statistical pattern-matching rather than reasoning β a foundational insight for anyone designing tests. Example: An LLM predicts the most statistically likely next word or phrase, not the most factually accurate one.
"They are trained on billions of words from different sources. Using deep learning, they make connections between all the words they are trained on to answer whatever questions we ask of them." β Demi Van Malcot
Generative AI
A category of AI model that produces new content: text, images, code, or other outputs, in response to input prompts, rather than returning a fixed or pre-programmed answer. LLMs are the most widely used type of generative AI. So what? Because outputs are generated fresh each time, the testing approaches used for traditional deterministic software don't transfer cleanly. Concepts like "expected result" and pass/fail need to be rethought. Example: ChatGPT, Claude, Bard, and Copilot are all generative AI applications built on large language models.
"It won't give an answer based on what is logically correct, but on what is statistically most likely. The sentences can be completely correct, while the answer is completely wrong." β Demi Van Malcot
Bias and fairness testing
Bias and fairness testing is a technique needed to test generative AI applications. It's goal is to check if the outputs of AI models is free from stereotypes, bias and discrimitory language. By doing bias and fairness tests while developing a generative AI application we ensure treats diverse inputs equitably and inclusively.
Comments
(11)
Want to practise API testing? Try an API that lies to you
As someone who is still hesitant about APIs (they are weird creatures I still don't fully comprehend π) this sounds frightening and intriguing at the same time
Ady asks the MoTaverse #2
For Dark and Aggressive I would put more then just Malevolant. For example glass half empty or doomsday thinking also counts as dark, but isn't really malevolant.
Also is Empirical included in Scientific mindset? And intuitive in Exploratory?
Ady asks the MoTaverse #1
I agree with Gary and Brandy, the first one immediately grabbed my attention and made me think it would be an interesting book.
AMA about transitioning to Quality Assistance model
What is for you the most difficult thing about doing quality assistance?
Announcement of new MoTaverse CEO
I am a fan of the snack-based incetive structure! πͺπ«ππ
Thanks for your contribution
1,000 songs in your pocket changed how the world listened to music...
Community Star
about 23 hours ago
Member joined your Chapter
Member joined TWiQ β This Week in Quality chapter
Community Star
1 day ago
Thanks for your contribution
1,000 songs in your pocket changed how the world listened to music...
Community Star
1 day ago
Thanks for your contribution
1,000 songs in your pocket changed how the world listened to music...
Community Star
1 day ago
Thanks for your contribution
1,000 songs in your pocket changed how the world listened to music...
Community Star
1 day ago
