Inference is the process of using a trained machine learning model to produce an output from new input data. In simple terms, training is where a model learns patterns, while inference is where the trained model applies what it has learned to make a prediction, classification, ranking, or generate content.
For example, when an AI model receives a prompt and generates a response, the generation process is inference. The same applies when an image model classifies an image, a recommendation system ranks products, or a fraud model predicts whether a transaction is suspicious.
From a testing perspective, inference is where the model's learned behaviour becomes observable and testable. Testing can examine output quality and accuracy, consistency, error handling, latency, throughput, and resource consumption across different inputs and operating conditions.
Inference testing becomes more challenging for generative and probabilistic models because identical inputs may produce different outputs depending on factors such as random seeds, sampling settings, model versions, and context. Tests may therefore need to evaluate properties and acceptable behaviour rather than rely entirely on exact expected outputs.
Inference also exposes distribution shift: production inputs may differ significantly from the data used during training or evaluation. For example, a fraud detection model may encounter transaction patterns that were rare or absent in its training data, causing its performance to degrade without any change to the model itself. Ongoing monitoring and representative production-like test data can help detect this behaviour.
A related risk is training-serving skew, where differences between the data processing or environment used during training and the one used during inference cause the model to behave differently in production. Differences in feature transformations, preprocessing, dependencies, or input formats can introduce this problem.
Performance is another important part of inference testing. Factors such as model size, input length, batch size, hardware, quantisation, and concurrency can affect latency, throughput, and resource usage. Online inference typically prioritises response-time objectives, while batch inference may place greater emphasis on throughput and processing efficiency.
A model that performs well during evaluation may still behave differently when exposed to real-world inputs and production conditions. Inference testing therefore needs to consider not just model accuracy, but also how the model behaves with changing data, workloads, dependencies, and operating environments.
