Alerts are defined
The service has alerts on symptoms its users would notice, each one actionable.
HighChecked automatically
- Applies to:
- Every service
- Rule id:
- observability.alerts-defined
Why it matters
Without alerts you learn about an outage from your users. With the wrong alerts the team is paged for things that need no action, and learns to ignore the pager. Both end in the same place: a real incident that nobody reacts to.
How to apply it
- Alert on symptoms: error rate, latency against the objective, queue lag. Causes (CPU, memory) belong on dashboards.
- Every alert that pages must need a human, now. If it can wait until morning it is a ticket.
- Tie alerts to the objective: page when the error budget is burning fast enough to run out.
- Link each alert to its runbook entry.
- Alerts are listed in the Alerts section of the MICROSERVICE.md.
Example
"Page when more than 2% of requests fail for 5 minutes. Ticket when p99 latency is above 800 ms for 30 minutes."
References
Checklists with this rule
Related agent skill: observability