Semantic caching stores responses by meaning rather than exact string match. For AI agents, the cache should consider tenant, task type, prompt version, tool state, embedding similarity, freshness window, and evaluator confidence before reusing an answer.
Where semantic caching fits
Semantic caching works best for repeated informational tasks: policy explanations, troubleshooting answers, classification, summarization, and routing decisions. It is risky for tasks that depend on rapidly changing account state or external side effects.
The cache should sit between the application and the model gateway. That placement lets it reuse answers across services while still enforcing tenant, prompt, and model boundaries.
- ->Cache stable knowledge responses.
- ->Avoid caching workflows that mutate external systems.
- ->Separate cache namespaces by tenant and prompt version.
- ->Attach evaluator confidence to every cache write.
Cache key design
A semantic cache key is more than an embedding. It also includes hard filters that must match before similarity is considered. Hard filters prevent a similar question from leaking across tenants, product areas, locales, or prompt versions.
Similarity only runs after the hard filters pass. This keeps the vector lookup small and makes cache behavior easier to explain in audits.
{
"tenant": "acme",
"workflow": "support.answer",
"locale": "en-US",
"prompt_version": "support-answer@18",
"model_family": "frontier-chat",
"embedding": "sha256:stored-vector-reference",
"fresh_until": "2025-03-01T00:00:00Z"
}Thresholds
A threshold that is too low creates wrong cache hits. A threshold that is too high misses the cost savings. Tune by workflow, not globally.
The right threshold depends on answer sensitivity. A marketing FAQ can tolerate broader reuse than an account-specific compliance explanation.
| Workflow | Suggested threshold | Reason |
|---|---|---|
| Generic FAQ | 0.86-0.90 | Low risk, high repetition |
| Troubleshooting | 0.90-0.94 | Needs symptom match |
| Policy guidance | 0.94-0.97 | Precise wording matters |
| Account actions | Do not cache | State changes too quickly |
Invalidation
Semantic caches need explicit invalidation rules. Prompt updates, policy changes, model migrations, and evaluator failures should all invalidate affected entries.
Store cache writes as trace events. When a cached answer causes a support issue, you need to know which request created it, which evaluator accepted it, and which prompt version produced it.
Evaluation before reuse
A cache hit is still a model decision from the user's point of view. Before an answer is reused broadly, it should pass the same quality checks as a fresh response for that workflow.
Vektor treats cache writes as evaluated artifacts. The original response, evaluator score, prompt version, source trace, and freshness window travel with the cache entry so reuse is explainable later.
| Check | Why it matters | Cache action |
|---|---|---|
| Evaluator pass | Prevents low-quality reuse | Write only when passing |
| Prompt version match | Avoids old instruction reuse | Invalidate on prompt change |
| Freshness window | Limits stale policy answers | Expire by workflow |
| Tenant boundary | Prevents cross-customer leakage | Hard namespace split |
Cache observability
A cache that only reports hit rate can hide quality regressions. Teams need to see avoided tokens, latency saved, evaluator pass rate, stale-hit rate, and incident rate for cached responses.
Vektor records cache lookup, hit, miss, write, invalidate, and evaluator outcomes as trace events. That lets a team inspect whether a faster answer was also a trustworthy answer.
- ->Track cache hit rate by workflow and tenant.
- ->Compare evaluator scores for cached and fresh responses.
- ->Alert when stale-hit rate rises after a policy update.
- ->Keep a source trace ID on every cache write.
Common questions
What is semantic caching?
Semantic caching reuses responses for requests with similar meaning, not only identical text.
Is semantic caching safe for AI agents?
It is safe for stable, read-only tasks when tenant boundaries, prompt versions, freshness windows, and evaluator checks are enforced.
How should cache hits be measured?
Measure hit rate, avoided tokens, latency saved, evaluator pass rate, and incident rate for cached responses.
Should cached answers run through evaluation?
The original answer should pass workflow-specific evaluation before it is written to the cache, and cached responses should be sampled in production for drift.
What invalidates a semantic cache entry?
Prompt changes, policy updates, source document changes, model migrations, evaluator failures, and expired freshness windows should invalidate affected entries.