Skip to main content

Trace a request in Grafana

Declaragent exposes Prometheus metrics by default (detached mode) and exports OpenTelemetry spans when you opt in. The testkit ships a docker-compose.yml with an OTel collector, Prometheus, Jaeger, and Grafana pre-wired.

Current span coverage

Tracing today covers the ingest and egress edges — three span types, each currently a root span (no parent/child links yet). Engine-turn, tool-call, and RPC spans plus cross-span trace propagation are on the roadmap. See Reference → Observability for the exact list.

Bring up the observability stack​

cd packages/testkit/observability
docker compose up -d
# Grafana on http://localhost:3000 (admin / admin)
# Jaeger UI on http://localhost:16686
# Prometheus on http://localhost:9090

See packages/testkit/alerts/README.md for the alert rules bundled with the stack.

Enable span export​

Install the OTel packages (they are not bundled with the CLI), then point Declaragent at the collector:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
declaragent up

The OTel service name is currently fixed to declaragent (OTEL_SERVICE_NAME is not read). See docs/OTEL_SETUP.md for details.

Explore​

Send a test message. In Jaeger (http://localhost:16686) → search service declaragent — you'll see the spans emitted today:

  • source.message — one span per inbound event-source message (message.id, message.topic, event.id, event.kind, correlation.id, outcome).
  • channel.outbound.send — one span per outbound channel send (conversation.id, content.kind, latency.ms).
  • channel.outbound.edit — one span per outbound message edit.

Each span stands alone — to follow one request end-to-end today, filter by the correlation.id attribute rather than expecting a single trace waterfall.

Alert rules​

The six shipped rule files live under packages/testkit/alerts/. Every alert carries a runbook_url that surfaces in this site's runbook index.