What AI-generated tests keep getting wrong (and what finally fixed it for us)

Full disclosure before anything else: I work at Testkube, and we launched a test creation feature today. I'll mention it near the end, but that's not really what this post is about.

What this post is about is a pattern I suspect many of you have seen up close.

Someone on the team asks an AI assistant for a test. The output looks great. Proper structure, sensible assertions, the right framework idioms. It gets a quick review, it gets merged, it goes green. And weeks later, when something breaks in exactly the area that test supposedly covered, someone opens it up and discovers it was asserting against its own mocks the whole time. It never touched the real system. It was never going to catch anything.

Here's the way I've come to think about why this keeps happening. A test is not general-purpose code. A test is a list of very specific claims about one particular system: this service name, this endpoint, this payload shape, this environment. General knowledge writes decent application code. It cannot write your tests, because almost nothing in your tests is general.

So when a model doesn't know your system, it does what models do: it fills the gaps with plausible guesses. And a plausible guess in a test is worse than an obvious mistake, because it survives review. The failure modes are always the same two. The test breaks for reasons that have nothing to do with the app, and someone burns an afternoon fixing generated code. Or the test passes forever, quietly, meaning nothing.

The thing that changed my mind about AI test generation wasn't a better model. It was moving the verification step. If a generated test runs against a real environment immediately, while it's still a draft, before anyone accepts it, then the guesses stop being dangerous. They get caught in seconds instead of in production. The test earns its way into your suite by actually running, not by looking right.

That principle holds no matter what tools you use, and if you take one thing from this post, take that: never accept a generated test that hasn't run against something real. Treat "it looks correct" as no information at all, because with generated code, it isn't.

The launch I mentioned: as of today, Testkube does this. You describe any test you need in plain language, it generates it in whatever framework you already use, Selenium, Playwright, you name it, runs it in your own infrastructure right away, and delivers it as a pull request to your repo. Any test type, including the load and infrastructure tests that most generation tools ignore.

If you want to see the whole argument played out live, our CTO Ole Lensmar and I are doing a session on October 8 

Happy to answer anything in the comments, including the skeptical questions. Those are usually the good ones 🙂