Skip to main content

Runbook: chaos DLQ rate breach

Alert source: packages/testkit/alerts/chaos-assertions.rules.yaml

Canonical markdown: docs/runbooks/chaos-dlq-rate-breach.md

Severity: critical (fires only during a chaos run).

Symptom​

During the current chaos run, DLQ volume exceeded 1% of ingress.

Likely cause​

  1. A fault exhausted the adapter's retry budget faster than the idempotency cache could absorb the replay.
  2. expire-idempotency-cache fired while a burst of retries was in flight.
  3. The adapter's nack semantics don't actually requeue under the injected fault (adapter bug).

Immediate mitigation​

# The chaos harness is the testkit chaos driver (packages/testkit/src/chaos)
# — stop the run by interrupting the harness process (Ctrl-C / SIGINT);
# the driver's stop() emits the run report on the way out.

Root-cause investigation​

# The chaos report is emitted by the driver's stop(); inspect the DLQ:
declaragent dlq list --kind source --source <id> --json

Post-incident​

  • Capture: DLQ sample, fault correlation, adapter + config under test.
  • Close when: a regression test reproduces the DLQ breach and a fix keeps the rate below 0.1%.
  • Post-mortem: mandatory — DLQ breaches under chaos block Phase-6 exit.