Diffusion models are a class of generative machine learning models that create new content by learning to reverse a gradual noise-adding process. They are commonly used to generate images, audio, video, and other forms of content.
During training, the model learns how content changes as noise is progressively added. During generation, it works in the opposite direction, starting with random noise and repeatedly removing it until a meaningful output is produced.
From a testing perspective, diffusion models introduce challenges that differ from traditional deterministic software. Their behaviour is stochastic, meaning that unless the random seed is fixed, the same prompt can produce different outputs across runs. Even with a fixed seed, changes to the model version, sampling settings, or execution environment can affect the result. Seed control can therefore support reproducible regression testing, while varied seeds can help explore the range of possible outputs.
Testers may assess output quality, consistency, robustness, safety, bias, and adherence to the input. These can be evaluated through approaches such as metamorphic testing, statistical evaluation, automated similarity metrics, and human evaluation.
Testing should also consider risks such as training-data memorisation, privacy or copyright exposure, adversarial inputs, prompt manipulation, and resource consumption. Performance testing is particularly relevant because generation quality, inference time, and computational cost can change significantly with factors such as output resolution and the number of denoising steps.
A diffusion model can produce an output that looks convincing while still being incorrect, unsafe, biased, or unexpectedly different from previous results. Testing therefore needs to evaluate not only whether an output was produced, but whether the behaviour and output are appropriate for the intended use.
