Skip to main content

Runbook: daemon bus pressure sustained

Alert source: packages/testkit/alerts/daemon.rules.yaml

Canonical markdown: docs/runbooks/daemon-bus-pressure-sustained.md

Severity: warning.

Symptom​

Bus in-flight count has held above 200 for 10 minutes. Source adapters with busPressure wiring will be continuously paused; non-wired sources may be blocking bus.publish on every call.

Likely cause​

  1. Downstream handler capacity shrank (scaled-in replicas, slow session).
  2. A subscriber is slow but not hung — publishes stay in-flight longer.
  3. A burst of legitimate traffic overwhelmed the configured capacity.

Immediate mitigation​

Raise the bus's high-watermark temporarily or scale handler capacity:

# Raise the source's concurrency/inflight limits in event-sources.yaml
# (limits.concurrency / limits.maxInflight), then restart to apply:
declaragent down && declaragent up -d

Root-cause investigation​

curl -s http://127.0.0.1:9464/status | jq
curl -s http://127.0.0.1:9464/metrics | grep -E 'source_inflight|source_messages'

Grafana: Daemon dashboard → Bus inflight + Publish rate panels.

Post-incident​

  • Capture: pressure peak, driving workload, mitigation (capacity vs threshold).
  • Close when: inflight < 50 for 15 minutes.