Production agent routing is the control plane that decides which provider, model, key, or fallback should handle each AI request. In Vektor, that decision should be captured on the trace with request metadata, budget pressure, model capability, provider health, latency, cost, and evaluation outcome.
When to route
Routing becomes useful as soon as one model is no longer enough. A support classifier, a code-generation agent, and a compliance reviewer should not all share the same model policy, latency target, or budget ceiling.
The safest pattern is to keep routing outside the application. Product code sends a normal chat, embedding, or tool request. The gateway adds policy: which provider to use, whether a fallback is allowed, and which limits apply to the calling team.
- ->Use task metadata for model capability decisions.
- ->Use headers or virtual keys for tenant and team policy.
- ->Use provider health and budget metrics for operational decisions.
- ->Record the final provider and model on every trace span.
Rule shape
A routing rule needs three pieces: a condition, a target, and a fallback policy. Conditions should be readable enough for review because routing errors become production behavior quickly.
Keep the first rules narrow. Route high-risk traffic explicitly, then let broad defaults catch ordinary requests. This keeps incident review simple when a provider outage or budget limit changes traffic shape.
{
"name": "route-reasoning-traffic",
"condition": "request.intent == 'reasoning' && team == 'research'",
"target": {
"provider": "anthropic",
"model": "claude-sonnet"
},
"fallbacks": [
{ "provider": "openai", "model": "gpt-4o" },
{ "provider": "google", "model": "gemini-pro" }
],
"limits": {
"max_latency_ms": 12000,
"monthly_budget_usd": 5000
}
}Fallbacks and retries
Fallbacks should protect availability without hiding quality changes. A provider timeout and a model-quality regression are different failures; routing should track both.
Retries are useful for transient network failures. Provider fallbacks are useful when the primary path is degraded. Model fallbacks are useful when the task can tolerate a smaller or cheaper model.
| Failure | Best response | Trace event |
|---|---|---|
| 429 rate limit | Retry with backoff or alternate key | provider.rate_limited |
| 5xx provider error | Fallback to healthy provider | provider.failed |
| Budget exhausted | Route to approved lower-cost model | budget.threshold_crossed |
| Quality regression | Block or send to human review | eval.failed |
How Vektor records routing
Routing decisions are most useful when they are visible next to the model span they affected. Vektor records the requested model, selected model, fallback chain, token count, latency, and cost on the same trace timeline.
That linkage lets teams compare quality before and after a routing change instead of treating provider selection as an invisible infrastructure detail.
Evaluation loop
Every routing change should be reviewed against quality, not only uptime. A cheaper model that passes format checks but fails policy accuracy can move cost out of the model bill and into support escalations.
Vektor keeps routing outcomes next to evaluator outcomes so teams can compare answer quality by provider, model, fallback reason, tenant, and prompt version. That makes routing a measurable release decision instead of an infrastructure hunch.
| Dimension | Question | Vektor signal |
|---|---|---|
| Model | Did quality change after the route? | eval_score by selected_model |
| Provider | Is one provider failing more often? | provider_error_rate |
| Tenant | Did one customer segment regress? | eval_score by tenant |
| Fallback | Did backup paths preserve quality? | fallback_used plus eval_result |
Operational checklist
Routing is safest when the rollout has explicit stop conditions. Before expanding traffic, teams should know which quality drop, latency increase, or cost threshold will pause the change.
The cleanest rollout starts with shadow traffic, then a small canary, then a gradual traffic increase. Each stage should compare routed requests against a baseline using the same evaluator suite.
- ->Define the baseline provider and model before the rollout starts.
- ->Attach fallback reasons to the trace, not only to logs.
- ->Compare routed and baseline traces on the same evaluation rubric.
- ->Pause expansion when quality, latency, or error budget crosses the release threshold.
Common questions
What is LLM gateway routing?
LLM gateway routing is the policy layer that selects the provider, model, key, and fallback path for each AI request.
Should routing live in application code?
For production systems, routing is easier to audit when it lives in a gateway or control plane instead of being duplicated across services.
What should be logged for every routed request?
Log the incoming intent, selected provider, selected model, fallback decisions, latency, token count, cost, and evaluation outcome.
How does Vektor compare routed requests?
Vektor groups traces by requested model, selected model, provider, fallback reason, tenant, and prompt version, then compares evaluator scores across those groups.
When should a fallback be blocked?
Block a fallback when the backup model cannot satisfy the task's safety, policy, latency, or data-residency requirements.