Observability

The agent runtime emits OpenTelemetry spans for every turn, LLM call, tool call, and durable step. Span names and attributes follow the GenAI semantic conventions. Compatible backends, including Langfuse, LangSmith, Arize Phoenix, and Datadog, can render token counts, models, and tool calls from those attributes.

Without the otel extra installed, observability hooks remain inactive.

Setup

pip install 'pyrula-agents[otel]'

Then one call at startup, before the worker starts:

from pyrula.agents.observability import configure

configure()

With no arguments, configure() honors the standard OTLP environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <token>"

That covers any OTLP backend, including a local collector. For the common SaaS backends there are shortcuts that fill in the endpoint and auth headers:

from pyrula.agents.observability import configure_langfuse

# reads LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST
configure_langfuse()

from pyrula.agents.observability import configure_langsmith

# reads LANGSMITH_API_KEY and LANGSMITH_PROJECT
configure_langsmith()

from pyrula.agents.observability import configure_datadog

# exports to the local Datadog Agent's OTLP intake (enable otlp_config in the Agent)
configure_datadog()

Keys can also be passed explicitly (configure_langfuse(public_key=..., secret_key=...)). To check what is being emitted without a backend, print spans to stdout:

configure(console=True)

Applications that already register an OTel tracer provider should not call configure(). Pyrula sends spans through the global provider.

Traced operations

One agent turn produces a span tree like:

invoke_agent billing_agent
├── chat claude-sonnet-4-6          gen_ai.usage.input_tokens=812 output_tokens=94
│   └── execute_tool lookup_invoice  gen_ai.tool.name=lookup_invoice
└── chat claude-sonnet-4-6          gen_ai.usage.input_tokens=931 output_tokens=187

Spans carry gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name, and token usage. Durable steps and stream batches get pyrula.* spans nested in the same trace.

Prompt and completion content

Message content (prompts, completions, tool arguments and results) is recorded on spans only when content capture is on. configure() leaves it off by default, following the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env var (off unless set). So pointing at an arbitrary OTLP backend doesn’t hand it prompts and completions unasked:

configure(capture_content=True)

configure_langfuse, configure_langsmith, and configure_datadog are prompt/completion viewers, so their shortcuts turn content capture on by default. Pass capture_content=False to any of them to turn it back off:

configure_datadog(capture_content=False)

Whatever gets recorded - captured content, and an exception message set as the span status - is secret-redacted first (API keys, tokens, passwords, credentials embedded in URLs, and the like). Captured values are truncated at 32 KB after redaction.

Metrics

configure() also exports the two GenAI client metrics:

• gen_ai.client.token.usage (histogram, by model, provider, and token type)
• gen_ai.client.operation.duration (histogram, seconds, tagged with error.type on failure)

Langfuse and LangSmith only ingest traces, so their shortcuts disable metrics export automatically. Pass metrics=False to configure() for any other trace-only backend.

Tracing across Kafka

KafkaAgentRunner propagates W3C trace context through Kafka headers, so a pipeline of agents on different topics shows up as one trace:

• Consuming a record starts a process {topic} consumer span, parented to the traceparent header stamped by the upstream producer.
• Records produced by the turn carry the processing span’s context, so the next consumer chains to this turn rather than skipping it.
• DLQ records keep the source record’s traceparent. From a dead letter you can jump straight to the trace of the run that failed.

None of this requires configuration beyond configure(). Without OTel, the runner falls back to forwarding the inbound traceparent header verbatim, so trace continuity survives a mixed fleet where only some services are instrumented.

For manual produce paths, build headers from the active span yourself:

from pyrula.agents.observability.otel import inject_current_context

headers = {k: v.encode() for k, v in inject_current_context().items()}

Conversations and usage in SQL

Pyrula can project the durable event log into two SQL tables for cross-run usage and conversation queries:

• pyrula_agent_messages - one row per conversation turn (role, JSONB content).
• pyrula_agent_usage - one row per LLM call: input_tokens, output_tokens, and the Anthropic cache breakdown (cache_creation_input_tokens, cache_read_input_tokens).

This is a read-only projection. The event log stays the system of record; the tables are derived from it and can be rebuilt at any time. Your agent code and ctx never read them, so there is no way for a query to affect a live run.

Enable projection

Pass project=True to the agent Postgres store. That runs the projection migrations once and registers the projectors; the worker drains new events into the tables every 60 seconds.

from pyrula.agents import PyrulaWorker
from pyrula.agents.stores.postgres import PostgresStore

store = PostgresStore(
    dsn="postgresql://localhost/pyrula",
    valkey_url="redis://localhost:6379",
    project=True,
)
worker = PyrulaWorker(agents=[support], store=store, llm=...)
worker.run()

The projection uses its own migration chain (pyrula_agent_alembic_version), distinct from the workflow archive’s (pyrula_archive_alembic_version). A host application’s own Alembic schema uses the default alembic_version, so all three can share a database without colliding.

Query the projection

Tokens per model, last 7 days:

SELECT model,
       SUM(input_tokens)  AS in_tok,
       SUM(output_tokens) AS out_tok,
       SUM(cache_read_input_tokens) AS cached_in_tok
FROM pyrula_agent_usage
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY model
ORDER BY in_tok + out_tok DESC;

Full transcript of one run, in order:

SELECT role, content
FROM pyrula_agent_messages
WHERE name = 'support' AND run_id = '<run-id>'
ORDER BY msg_seq;

Cost

Pyrula records tokens, not money. Rates vary by provider, tier, batch mode, and cache class, and may change between library releases. Multiply the token columns by current rates, or use Pyrula Cloud’s project-level price table.

The same split holds in tracing: the OpenTelemetry spans carry gen_ai.usage.* token attributes and no cost, so any backend that prices them (Langfuse, LangSmith, Datadog) does it with its own rates.

Backfill

Runs archived before you turned projection on are filled in with pyrula reproject:

pyrula reproject                                 # every archived run, every agent
pyrula reproject --agent support                 # one agent
pyrula reproject --agent support --run-id <id>   # one run (--run-id needs --agent)