SWE-bench Longitudinal: Temporal Memory Beats RAG on Real Code Histories
On 130 marker-free scenarios extracted from SWE-bench Lite/Verified buggy→fixed histories, MemStrata temporal_v6 hits 0.908 accuracy vs ~0.57 RAG — with stale-fact-error 0.023 vs 0.262.
From synthetic templates to real GitHub evolution
Paper 1 proves the mechanism on carefully constructed evolving benchmarks. Paper 2 asks the harder product question: does temporal validity hold on real software evolution?
We extract 130 clean single-value, marker-free scenarios from SWE-bench Lite and Verified (707 raw records → filtered atomic value changes). Each scenario ingests buggy then fixed state; the question targets the current value with no textual currency markers.
Headline numbers (same local 7B stack)
On the longitudinal suite:
- Accuracy: temporal_v6 0.908 · naive RAG 0.569 · advanced RAG (LLM rerank) 0.585
- Stale-fact-error: temporal_v6 0.023 · both RAG variants 0.262
- Mean retrieval latency: ~2.1s vs ~21s for advanced RAG
- Active memory size: 135 facts vs 260 for RAG (bounded growth on contradictions)
Reranking with the same weak local model barely moves accuracy while multiplying latency ~10× — another data point that “more LLM on the read path” is not a free lunch.
What this means for coding agents
Real repos rename functions, bump dependencies, and migrate configs continuously. An assistant that remembers last week’s signature is a liability. Temporal validity is the difference between compounding project knowledge and replaying git history out of order.
MemStrata is designed for that reality: local-first, deterministic supersession, and honest metrics you can see on the dashboard.