Retrieval and memory evaluations measure whether an AI agent selected the right context and used it correctly. The core checks are retrieval precision, retrieval recall, source freshness, answer grounding, contradiction detection, and missing-context behavior.
What to measure
Retrieval quality is not only about whether the answer is correct. A model can answer correctly from prior knowledge while ignoring the retrieved context, which is dangerous in private or fast-changing domains.
The evaluation should separate retrieval from generation. First ask whether the right context was selected. Then ask whether the answer used that context faithfully.
| Metric | Question | Failure mode |
|---|---|---|
| Precision | Were selected chunks relevant? | Distracting context |
| Recall | Were required facts retrieved? | Missing evidence |
| Freshness | Was stale memory avoided? | Outdated answer |
| Grounding | Did the answer cite retrieved facts? | Hallucination |
Dataset design
A retrieval eval dataset should include queries, required sources, irrelevant near-matches, expected answer behavior, and freshness constraints. Near-matches are important because they reveal whether embeddings are over-broad.
For memory systems, include temporal cases. The newest fact should override older facts, but old facts may still matter for history and preference modeling.
- ->Include positive and negative source examples.
- ->Mark required chunks separately from helpful chunks.
- ->Add stale-memory cases for long-running agents.
- ->Keep expected behavior distinct from exact wording.
Judge rubric
The judge should score whether the answer is supported by retrieved context, not whether it sounds plausible. This distinction matters when the model already knows a public fact but the workflow requires private evidence.
Ask for contradiction labels as well as quality scores. A grounded but incomplete answer is usually safer than a confident answer that contradicts the source.
criteria:
- answer_uses_required_sources
- answer_does_not_contradict_context
- answer_marks_missing_context_when_needed
- answer_ignores_irrelevant_near_matches
labels:
contradiction: block
missing_required_source: review
fully_grounded: passProduction loop
Offline retrieval evals are not enough because knowledge bases drift. Track live retrieval failures, user corrections, low-confidence answers, and evaluator misses back into the dataset.
The best memory systems create a loop: traces identify retrieval failures, evaluations classify the failure, and dataset updates prevent the same failure from returning.
Memory conflict handling
Long-running agents often carry contradictory memories: an old preference, a newer correction, and a policy document that changed last week. The evaluator should check whether the agent selected the freshest relevant memory and handled conflicts explicitly.
Vektor marks memory writes with source, timestamp, store type, and version. That lets the evaluation distinguish a retrieval miss from a stale-memory error or a generation error.
| Conflict | Expected behavior | Failure label |
|---|---|---|
| Old preference vs new correction | Use the newer correction | stale_memory |
| Private note vs policy source | Prefer policy for compliance | source_priority |
| Missing required context | Ask for clarification | missing_context |
| Contradictory retrieved chunks | Surface uncertainty | unsupported_claim |
Evaluation report
A useful retrieval report should explain both retrieval and generation behavior. It should show which chunks were required, which were retrieved, which were used in the answer, and which claims were unsupported.
Vektor stores that report beside the trace so product teams can review failures without reconstructing the retrieval state by hand.
- ->Show required sources, retrieved sources, and omitted sources separately.
- ->Label stale memory, irrelevant retrieval, contradiction, and unsupported claims.
- ->Connect each answer claim to supporting context when available.
- ->Promote recurring failures into the regression dataset.
Common questions
What is a retrieval evaluation?
It measures whether the retrieval system selected the right context for an AI answer.
What is a memory evaluation?
It measures whether an agent stores, retrieves, updates, and uses memory correctly across time.
How do you evaluate grounded answers?
Check whether the answer uses required sources, avoids contradiction, cites current context, and admits missing information when context is incomplete.
How should stale memory be evaluated?
Include temporal cases where an older memory conflicts with a newer correction, then verify that the agent uses the freshest relevant fact.
What should a retrieval evaluation report include?
It should include required sources, retrieved sources, used sources, omitted sources, unsupported claims, freshness labels, and trace links.