Skip to main content

oncall-escalator template

Alertmanager webhook → Claude triage → Slack DM. Demonstrates the webhook source with HMAC verification, X-Alertmanager-Fingerprint as an idempotency key, and the SendMessage tool on an outbound-only channel.

Scaffold​

declaragent init --template oncall-escalator --provider anthropic

Files produced:

./
├── agent.yaml
├── event-sources.yaml       # webhook source + HMAC secret ref
├── channels.yaml            # outbound Slack DM
├── skills/triage.md
├── fixtures/mock-alert.json
├── .env.example
└── README.md

Canonical starter: templates/oncall-escalator/.

Configure​

  1. Generate a webhook HMAC secret and store it in Vault / AWS SM / env: 
    ${secret:oncall/webhook-hmac}
  2. Point Alertmanager at https://<your-host>/webhook/alertmanager.
  3. Create a Slack app with chat:write + im:write and install it to the workspace.

Run locally​

declaragent up -d
# simulate an alert (webhook listener binds port 7777 by default):
SIG=$(openssl dgst -sha256 -hmac "$ALERTMANAGER_WEBHOOK_SECRET" mock-alert.json | awk '{print $NF}')
curl -X POST http://localhost:7777/webhook/alertmanager \
  -H "X-Alertmanager-Fingerprint: test-123" \
  -H "X-Alertmanager-Signature: sha256=$SIG" \
  -H "X-Alertmanager-Timestamp: $(date +%s)" \
  --data-binary @mock-alert.json
declaragent logs -f        # watch the triage + DM fire

The agent triages the alert (reads the runbook links, checks severity) and DMs the on-call engineer on Slack.

Key points​

  • Idempotency. Alertmanager retries on 5xx. The X-Alertmanager-Fingerprint header doubles as the idempotency key — duplicate fingerprints inside the dedup window (10-minute in-memory cache, 24-hour store) are dropped at ingress.
  • HMAC. The webhook source verifies the X-Alertmanager-Signature header (sha256=<hex> over the raw body) against the configured secret, and rejects requests whose X-Alertmanager-Timestamp drifts outside the 5-minute replay window. Missing or mismatched signature → 401 EUNAUTHORIZED.
  • Outbound-only channel. The Slack channel is configured with direction: outbound — inbound @mentions are ignored.