Skip to main content

Runbook: secret access denied spike

Alert source: packages/testkit/alerts/security.rules.yaml

Canonical markdown: docs/runbooks/secret-access-denied-spike.md

Severity: critical.

Symptom​

Tenant {{ tenant_id }} is seeing > 0.1 denials/s on secret-access requests for 5 minutes. Downstream adapters that need the secret will fail on every invocation.

Likely cause​

  1. Misconfigured secret ref in a recently-deployed skill — retry loop is hammering the denial path.
  2. Secret path violates the tenant's configured scope (Phase-6 isolation check).
  3. Someone is probing the secret-scope boundary for reconnaissance.

Immediate mitigation​

If a specific ref is the hot one, block it at the secret-provider layer:

# There is no secrets-block verb — revoke the credential at the
# provider (Vault policy / IAM), rotate, and restart:
declaragent secrets rotate <ref>
declaragent down && declaragent up -d

Revoking at the provider short-circuits the denial loop at its source.

Root-cause investigation​

# Top denied refs and requesters:
declaragent audit query --tenant <tenant_id> --kind secret_access --outcome denied --since -30m --json

# Verify tenant scope config:
declaragent tenants show <tenant_id>

If the requester is a skill / plugin, inspect its manifest — a config change may have introduced an out-of-scope ref.

Post-incident​

  • Capture: top 5 denied refs, requesters, scope config at time of incident.
  • Close when: denial rate < 0.01/s for 30 minutes.
  • If reconnaissance is suspected, file a security review ticket with the platform team.