Agent observability is the practice of tracing every meaningful step in an AI workflow, not just the final model call. A useful trace includes prompt versions, model parameters, tool inputs, tool outputs, memory reads, routing decisions, costs, and evaluation results.
Trace model
A single request can contain a planner span, several model spans, multiple tool spans, a memory lookup, a final response, and a post-hoc evaluator. If those steps are logged as one flat blob, the failure path is hard to recover.
The span hierarchy should mirror the agent hierarchy. Parent spans represent user-visible jobs. Child spans represent model calls, tool calls, retrieval, and guardrail checks.
trace.start("support-agent.reply", {
user_id: "usr_123",
workflow: "refund-resolution"
});
trace.span("planner.select_next_action", { model: "gpt-4o-mini" });
trace.span("memory.retrieve", { store: "episodic", results: 6 });
trace.span("tool.call", { name: "orders.lookup", status: "ok" });
trace.span("model.generate", { model: "claude-sonnet", tokens: 1240 });
trace.span("eval.score", { rubric: "policy_accuracy", score: 0.94 });Minimum fields
Every model span should preserve enough context to replay the decision. That does not mean logging sensitive raw data forever. It means storing prompt versions, parameter hashes, tool names, and redacted payload references.
For enterprise systems, redaction should happen before persistence. Keep a pointer to the secure payload store when the raw prompt cannot be retained in the trace database.
| Field | Why it matters | Example |
|---|---|---|
| prompt_version | Explains behavior changes | support-agent@42 |
| model | Separates provider drift from prompt drift | gpt-4o-mini |
| tool_name | Shows external side effects | crm.update_case |
| memory_store | Identifies retrieval source | semantic |
| eval_score | Connects trace to quality | 0.91 |
Debug loop
A good trace shortens the path from customer report to root cause. Start with the final response, jump to the evaluator that marked it wrong, then inspect the earliest span that introduced the bad assumption.
Most agent bugs are not in the last model call. They come from a missing retrieved fact, a malformed tool result, a stale prompt version, or a routing decision that moved the task to a weaker model.
- ->Compare passing and failing traces for the same workflow.
- ->Filter by prompt version before changing prompts.
- ->Inspect tool result schemas before blaming the model.
- ->Attach evaluator feedback to the exact span that caused it.
Operational metrics
Latency, cost, and quality should be viewed together. A model swap that lowers cost but increases rework may be more expensive once downstream retries and human reviews are counted.
The most useful dashboard groups traces by workflow, tenant, prompt version, provider, and model. Those dimensions turn individual failures into patterns.
Privacy and redaction
Full-span tracing does not require storing every raw prompt forever. Sensitive payloads can be redacted before persistence while the trace keeps stable references, hashes, prompt versions, model settings, and evaluator outcomes.
The important distinction is between replayable engineering context and sensitive user content. Vektor preserves enough structure to debug behavior while allowing payload storage policies to vary by customer, tenant, or workflow.
- ->Redact secrets and personal data before trace persistence.
- ->Store prompt versions and payload hashes even when raw payloads are excluded.
- ->Keep tool names, status codes, and schema errors visible in the trace.
- ->Use tenant-level retention controls for regulated workflows.
Replay and regression analysis
The most valuable trace is one that can become a regression case. When a production failure is confirmed, the trace should be convertible into a dataset item with the original inputs, expected behavior, and evaluator rubric.
Replay also separates prompt failures from provider drift. If the same trace passes with one model and fails with another, the release decision is different from a prompt bug that fails everywhere.
| Replay mode | Use case | Output |
|---|---|---|
| Same prompt, same model | Confirm reproducibility | Stable failure or transient failure |
| Same prompt, new model | Model migration review | Quality delta |
| New prompt, same model | Prompt release review | Behavior delta |
| Baseline vs canary | Production rollout | Release recommendation |
Common questions
What is full-span tracing for AI agents?
Full-span tracing records every major step in an agent workflow, including model calls, tools, memory, routing, and evaluations.
How is agent tracing different from normal logging?
Agent tracing preserves the causal structure of a workflow, while normal logs often store events without parent-child relationships.
Which trace fields matter most for debugging?
Prompt version, model, tool input and output, memory source, routing decision, latency, cost, and evaluation score are the most important fields.
Can traces be useful without raw prompts?
Yes. Prompt versions, parameter hashes, redacted payload references, tool metadata, status codes, and evaluator outcomes are often enough to diagnose production changes.
How should a failed trace become a regression test?
Attach the failed trace to a dataset item with the original context, expected behavior, prohibited behavior, and the evaluator that caught the failure.