Alerts are defined

The service has alerts on symptoms its users would notice, each one actionable.

HighChecked automatically

Applies to:
Every service
Rule id:
observability.alerts-defined

Why it matters

Without alerts you learn about an outage from your users. With the wrong alerts the team is paged for things that need no action, and learns to ignore the pager. Both end in the same place: a real incident that nobody reacts to.

How to apply it

  • Alert on symptoms: error rate, latency against the objective, queue lag. Causes (CPU, memory) belong on dashboards.
  • Every alert that pages must need a human, now. If it can wait until morning it is a ticket.
  • Tie alerts to the objective: page when the error budget is burning fast enough to run out.
  • Link each alert to its runbook entry.
  • Alerts are listed in the Alerts section of the MICROSERVICE.md.

Example

"Page when more than 2% of requests fail for 5 minutes. Ticket when p99 latency is above 800 ms for 30 minutes."

References

Checklists with this rule

Related agent skill: observability

  • An availability objective is set

    The service has a stated availability target, so "is it reliable enough?" has an answer.

    HighChecked automatically

  • Consumer lag is monitored

    The service measures how far behind its consumers are and alerts before the delay hurts users.

    HighChecked by hand

  • A p99 latency target is set

    The service states how fast its slowest normal requests should be.

    MediumChecked automatically

  • Distributed tracing is on

    Requests carry a trace context through the service, so one request can be followed across services.

    MediumChecked automatically

  • A dashboard exists

    The service has a dashboard showing its traffic, errors and latency at a glance.

    LowChecked automatically