Distributed tracing is on

Requests carry a trace context through the service, so one request can be followed across services.

MediumChecked automatically

Applies to:
Every service
Rule id:
observability.traces-enabled

Why it matters

In a distributed system the question is rarely "is this service slow?" but "where did this request spend its time?". Logs and metrics from one service can't answer it. A trace can, but only if every service on the path passes the context along. One service that drops it cuts the trace in two.

How to apply it

  • Propagate the W3C traceparent header on every inbound and outbound call, and through message headers for events.
  • Create a span around each outbound call and each significant unit of work.
  • Put the trace id in every log line, so you can jump from a log to the trace.
  • Sample: all errors, and a fraction of the successes.

Example

A slow checkout shows one trace: gateway 12 ms, checkout 40 ms, payments 1.9 s, of which 1.8 s is a call to the fraud service.

References

Checklists with this rule

Related agent skill: observability

  • Alerts are defined

    The service has alerts on symptoms its users would notice, each one actionable.

    HighChecked automatically

  • An availability objective is set

    The service has a stated availability target, so "is it reliable enough?" has an answer.

    HighChecked automatically

  • Consumer lag is monitored

    The service measures how far behind its consumers are and alerts before the delay hurts users.

    HighChecked by hand

  • A p99 latency target is set

    The service states how fast its slowest normal requests should be.

    MediumChecked automatically

  • A dashboard exists

    The service has a dashboard showing its traffic, errors and latency at a glance.

    LowChecked automatically