How do you test that something is AI generated?

May 22, 2025

Create Memory
One man is presenting on a large stage. Behind him, there is a huge screen. A host is standing to the right of the stage.
In this memory: Jarsto van Santen

"There are hints. There are signatures. There are probabilities that are unique to this set of data. The same thing goes for something like ChatGPT. For the different GPT variants, they have statistical correlations which you can work out probability-based. The idea was that in post-processing, we force the AI to prefer certain synonyms that humans tend not to prefer. So you generate a fingerprint which says — and again this is probabilistic — if you have these choices in language, if you have these coincidences, these words linking up in certain frequencies, that makes it very likely this was AI generated."

Brighton, September 2024 at TestBash. Jarsto van Santen answers a question from the audience after introducing how Generative AI works.

Talk available at: https://www.ministryoftesting.com/testbash-sessions/how-generative-ai-works-a-very-rough-guide

Community Lead at Ministry of Testing
he/him
Hello, I'm Simon. Since 2003 I've had various roles in testing, tech leadership and coaching. I believe in the power of collaboration, creativity and community. 🎓 MoT-STEC qualified.
Sign in to comment
TestRail 9.5: AI That Truly Understands Testing image
TestRail AI understands testing to help reach your goals faster with true intelligence. Join us on Sep 18 to learn more.
Explore MoT
Xray AI in action: Test Case & Model Generation for modern QA teams image
Wed, 17 Sep
with Ivan Fillipov, Solution Architect at Xray
Advanced prompting for testers image
Advanced prompting skills to turn AI into your trusted testing companion.
Leading with Quality
A one-day educational experience to help business lead with expanding quality engineering and testing practices.
This Week in Testing image
Debrief the week in Testing via a community radio show hosted by Simon Tomes and members of the community
Subscribe to our newsletter
We'll keep you up to date on all the testing trends.