Consumer lag is monitored

The service measures how far behind its consumers are and alerts before the delay hurts users.

HighChecked by hand

Applies to:
Services that consume events
Rule id:
observability.consumer-lag-monitored

Why it matters

A consumer that has stopped looks healthy from the outside: no errors, no failed requests, just data that gets older. Lag is the one signal that shows it. Without it the first report comes from a user asking why their order from an hour ago is still pending.

How to apply it

  • Export lag for every consumer group or subscription: messages behind, and better, the age of the oldest unprocessed message.
  • Alert on age, not on count. A thousand messages can be one second or one hour of work.
  • Set the threshold from what users can tolerate, and page only when it is crossed for several minutes.
  • Put lag on the service dashboard beside throughput and error rate.
  • Watch the dead-letter queue the same way.

Example

Alert: "shipping consumer: oldest unprocessed OrderPlaced is older than 5 minutes, for 10 minutes". The dashboard shows lag per partition, messages per second and dead-letter count.

References

Checklists with this rule

Related agent skill: observability

  • Alerts are defined

    The service has alerts on symptoms its users would notice, each one actionable.

    HighChecked automatically

  • An availability objective is set

    The service has a stated availability target, so "is it reliable enough?" has an answer.

    HighChecked automatically

  • A p99 latency target is set

    The service states how fast its slowest normal requests should be.

    MediumChecked automatically

  • Distributed tracing is on

    Requests carry a trace context through the service, so one request can be followed across services.

    MediumChecked automatically

  • A dashboard exists

    The service has a dashboard showing its traffic, errors and latency at a glance.

    LowChecked automatically