Prompt versioning is the practice of assigning stable versions to prompts and the runtime settings that affect AI behavior. A useful prompt release includes the prompt text, model, parameters, tool catalog, retrieval policy, evaluation dataset, rollout state, and rollback target.
Version boundary
A prompt is rarely just text. Behavior changes when the model changes, when temperature changes, when a tool is added, when retrieval context changes, or when a system instruction is moved.
The version boundary should include every setting that can change output. That makes regressions explainable because each trace can point back to the exact release artifact that produced it.
- ->Version system, developer, and user prompt templates.
- ->Store model, parameters, and provider routing policy.
- ->Record tool catalog and retrieval configuration.
- ->Attach evaluation suites and pass thresholds.
Release record
A release record is the audit object for an AI behavior change. It should answer who changed the prompt, why it changed, which datasets passed, and where the prompt is currently deployed.
Release records make production review cleaner because they connect prompt versions, model behavior, evaluation results, and trace evidence in one place.
{
"prompt_id": "support-answer",
"version": 43,
"model": "gpt-4o-mini",
"temperature": 0.2,
"tools": ["orders.lookup", "policies.search"],
"retrieval_policy": "support-kb@12",
"eval_suite": "support-regression@8",
"rollout": { "state": "canary", "traffic_percent": 10 },
"rollback_to": "support-answer@42"
}Rollouts
Do not move all production traffic to a new prompt at once. Start with offline evals, run a shadow pass against recent traffic, then canary a small percentage of real requests.
Canaries should watch both quality and operations. A prompt that improves helpfulness but doubles tool calls may still be a bad release.
| Stage | Traffic | Exit criteria |
|---|---|---|
| Offline eval | 0% | Regression suite passes |
| Shadow | Mirrored only | No severe judge failures |
| Canary | 5-10% | Quality stable, cost acceptable |
| General | 100% | Rollback target retained |
Rollback
Rollbacks should be boring. Keep the previous prompt version warm, preserve its model settings, and make sure routing rules still allow it.
When an incident happens, the trace should show the active prompt version and the previous known-good version. That one field can save hours of diagnosis.
Canary metrics
A prompt canary needs both quality and operational metrics. A release that improves tone but increases tool calls, retrieval misses, or manual escalations may still be worse for the business.
Vektor compares canary and baseline traffic across evaluator score, latency, token cost, fallback rate, tool error rate, and escalation rate. That gives release owners a broader view than a single pass/fail metric.
| Metric | Why it matters | Release action |
|---|---|---|
| Evaluator score | Measures answer quality | Expand, hold, or block |
| Tool calls per task | Shows workflow efficiency | Investigate prompt planning |
| Escalation rate | Captures user-facing failures | Hold rollout |
| Fallback rate | Shows routing instability | Review model/provider choice |
Approval history
Prompt releases should leave an audit trail: who changed the prompt, which evaluation suite passed, who approved the rollout, and which rollback target was available.
The audit trail is especially useful when prompts are edited outside the application deploy cycle. Vektor connects prompt versions to traces so teams can see exactly which behavior was live for a given request.
- ->Record author, reviewer, release owner, and approval timestamp.
- ->Attach the passing evaluation run to the release record.
- ->Keep a rollback target until the new version is stable.
- ->Link production incidents back to the active prompt version.
Common questions
What should be included in a prompt version?
Include prompt text, model, parameters, tools, retrieval policy, evaluation suite, rollout state, and rollback target.
Why version prompts separately from code?
Prompt changes can alter production behavior without code changes, so they need their own release, review, and rollback process.
How do traces help prompt releases?
Traces connect each output to the exact prompt version, model settings, tools, and evaluations used for that request.
What metrics should a prompt canary watch?
Watch evaluator score, latency, token cost, tool-call count, fallback rate, escalation rate, and user correction rate.
How long should rollback targets be retained?
Keep the previous known-good version available until the new prompt has passed canary traffic and production monitoring for the workflow's risk window.