Marker-Free Benchmarks for Evolving Knowledge: Stop Letting Labels Inflate RAG Scores
If your evolving-knowledge benchmark marks outdated chunks as “[OUTDATED]”, baselines win by reading the label. MemStrata’s marker-free protocol removes that contamination — and scores drop up to 14 points when markers are removed.
The contamination nobody notices
Evaluating memory under knowledge evolution is easy to cheat by accident. If a stale fact carries a textual marker — “legacy”, “deprecated”, “[OUTDATED]” — a plain retrieval baseline can disambiguate by reading the label rather than by any temporal mechanism. Scores go up. The science goes down.
We enforce a strict marker-free invariant: in every evolving benchmark, the stale and current versions of a fact are textually identical except for the changed value. No old/new framing. The only signal of currency is ingestion order — which only a temporal mechanism can exploit.
Four evolving suites + forced-answer supplements
Paper 1 evaluates on two static suites (project-fact QA, multi-session dialogue) and four marker-free evolving suites: code mutation, configuration migration, dependency bumps, and API evolution. We also run a forced-answer supplement that disables abstention on RAG conditions to expose the stale-commitment that polite abstention otherwise hides.
Removing a marker from an earlier contaminated version of a benchmark dropped baseline accuracy by up to 14 points — confirming the effect is real and large enough to invert research conclusions.
What we recommend for the field
If you publish agent-memory results on evolving knowledge:
- State whether your benchmark is marker-free (by test).
- Report stale-fact-error rate separately from accuracy.
- Report forced-answer behavior; abstention can mask structural failure.
- Run competitors on the same local model and harness when claiming architectural wins.
We release harness, prompts, and datasets with the paper so others can reproduce the protocol.