The Value-Change Ceiling: Only ~19% of Real Code Fixes Are Clean Supersessions
Across 707 SWE-bench fixes, deterministic temporal memory natively covers ~19% of evolution. Contract extraction lifts to ~24%. The remaining ~76% is logic — and overclaiming extraction is a research risk.
Scope is a first-class research result
After Papers 1–2, the natural question is not “does supersession work?” but “on how much of real code evolution does it apply?” Paper 3 answers with a taxonomy over 707 real GitHub fixes (SWE-bench), frozen annotation rules, and both LLM + human validation.
Headline: ~19% native value-change ceiling. A bounded contract extractor (params, arity, return type) lifts the extraction ceiling to only ~24% overall. Full-file AST extraction across declined fixes recovers just 6.6% additional contract changes — not a flood of hidden triples.
The majority is logic — and that is okay
About 76% of fixes are logic/control-flow that do not reduce to a clean (subject, relation, object) triple. Pretending otherwise would inflate claims and corrupt the product. MemStrata’s research stance is deliberate: own the CERTAIN value-change spine perfectly, then handle the rest with detect-and-flag (Paper 4) rather than hallucinated structure.
Human validation on a stratified 82-fix sample (Cohen’s κ = 0.144 on fine 5-way labels) shows disagreement concentrates off the load-bearing axis: the native value rate remains stable even when membership shifts. Two bounds survive: the ~19% ceiling and logic majority.
Recency is not a temporal model
Controlled experiments show recency-aware RAG — including LLM “pick the newest chunk” rerankers — fails to match the ledger. A recency-adversarial re-read of an old value exposes a sharp boundary where both recency and a naive ledger can resurrect stale state without additional guards.
For teams evaluating memory products: ask for taxonomy coverage, not just a single accuracy number on cherry-picked value flips.