Flaky Test

A flaky test refers to a test that sometimes passes and sometimes fails without changes to the code. These tests are unreliable and can make debugging difficult.

A test that passes in one run and fails in another — with zero changes to the code, the environment, or the test itself.

That non-determinism is what makes it flaky. It's unreliable not because the application changed, but because something about the test is inherently unstable — a race condition, a timing dependency, reliance on external state, etc.

Calling a test "flaky" just because it caught a bug muddies the conversation and, worse, can lead teams to dismiss legitimate test failures.

A test that produces inconsistent results across runs without any change to the code under test, passing sometimes and failing at other times. Flakiness is commonly caused by timing issues, asynchronous waits, unstable test data, or dependencies on external services. In an AI-assisted workflow, flaky tests form one of the key failure clusters to identify after a test run, and they require human judgement to distinguish from genuine product defects. Automatically patching or suppressing a flaky test without understanding its root cause risks masking a real problem in the system.