A p99 latency target is set

The service states how fast its slowest normal requests should be.

MediumChecked automatically

Applies to:
Every service
Rule id:
observability.latency-target

Why it matters

Averages hide the requests that hurt. The p99 is what one user in a hundred gets, and in a chain of calls most users hit somebody's p99. A stated target lets callers set their timeouts from a fact and lets you alert before "slow" turns into "down".

How to apply it

  • Set the target on a high percentile (p99, or p95 for low-traffic services), per endpoint class if they differ a lot.
  • Measure it at the server and, where you can, at the caller.
  • Budget it: your target has to fit inside your callers' timeouts, and your dependencies' targets have to fit inside yours.

Example

slo.latencyP99: 300 ms for reads and 800 ms for writes, with the gateway timeout at 2 s.

References

Checklists with this rule

Related agent skill: performance-scalability

  • Alerts are defined

    The service has alerts on symptoms its users would notice, each one actionable.

    HighChecked automatically

  • An availability objective is set

    The service has a stated availability target, so "is it reliable enough?" has an answer.

    HighChecked automatically

  • Consumer lag is monitored

    The service measures how far behind its consumers are and alerts before the delay hurts users.

    HighChecked by hand

  • Distributed tracing is on

    Requests carry a trace context through the service, so one request can be followed across services.

    MediumChecked automatically

  • A dashboard exists

    The service has a dashboard showing its traffic, errors and latency at a glance.

    LowChecked automatically