Cosine Similarity Cannot Detect Contradictions (AUROC 0.59): What That Means for Agent Memory
Calibration results show cosine similarity cannot separate contradicted facts from duplicated ones. Contradictions are more embedding-similar than rephrases. Here is the data and the design implication.
The calibration experiment
If your agent memory uses “if similar enough, treat as update” logic, you are betting that cosine space separates three relationships: duplicate, contradict, and novel. We measured that bet on 98 labeled pairs (32 duplicate, 22 merge, 22 contradict, 22 novel) with nomic-embed-text (768-d).
Result: cosine AUROC for separating duplicates from the rest is 0.5926. Mean cosine for contradictions: 0.8119. Mean for duplicates: 0.7998. The value flip is a smaller edit than a paraphrase — so the “contradiction” sits closer to the original than a restatement does.
Why thresholds and learned judges fail
Maximum precision achievable at any duplicate threshold is 0.667. You cannot reach a 0.95 safety floor. Learned classifiers on top of similarity (our gate-judge conditions) remain unreliable; in abstention regimes they can leak stale facts 25–60% of the time.
LLM relevance verifiers also do not solve staleness: they have no temporal signal and add ~8× latency. The structurally correct mechanism is deterministic keying, not a smarter embedding distance.
Design implication for AI coding tools
Memory systems for coding agents should key mutable facts by structured identity — (entity, relation) or symbol+attribute — and apply supersession on object change. Embeddings remain excellent for retrieval ranking of active facts; they are the wrong tool for write-time contradiction detection.
That split — vector recall under a deterministic CERTAIN spine — is the architecture MemStrata ships, and the reason our stale-fact-error rate collapses while static recall holds.