Symptom-based alerting is an approach to production alerting where alerts fire on problems users actually experience, such as failed journeys or slow responses, rather than on internal causes such as high CPU or memory usage. It is sometimes described as alerting on symptoms, not causes, and is a common way to reduce noise in monitoring.
Why it matters: A server running at 90 percent CPU while users have a flawless experience is just a data point, not an emergency worth waking someone for. Alerting on symptoms keeps attention on user experience and business impact. It also guards against alert fatigue, because each alert points to something that genuinely needs a response.
In practice: Teams start by asking what failure looks like for the user and how they would know it has happened. They then build metrics and synthetic monitors around key journeys, such as sign-in, search or checkout, and alert when those degrade. Cause-level metrics such as CPU or queue depth are still collected, but they are used for investigation through observability tools rather than for paging people.
Examples:
- Alerting when the share of completed checkouts drops sharply, rather than when a database server reaches a memory threshold.
- Paging the on-call engineer when a synthetic monitor of the login journey fails repeatedly from several locations.
- Raising an alert when basket abandonment rises above its usual level for an hour.
Common pitfalls:
- Treating green infrastructure metrics as proof that everything is fine, when users may still be unable to complete key journeys.
- Choosing symptoms so broad that an alert fires without giving any clue which feature or journey is affected.
Does symptom-based alerting mean ignoring infrastructure metrics? No. Infrastructure metrics are still useful for diagnosing why something broke. They just are not the main trigger for waking people up.
How can testers help with symptom-based alerting? By bringing questions about how a feature will be known to be working in production into refinement, so user-focused metrics and alerts are part of the definition of done.
