Skip to content
Computing Life · Share · Yage

When a model gets a fact wrong, first figure out if it never learned it or just can't recall it this time

Google Research's ICML 2026 paper tested 13 models on 2,150 WikiProfile facts. A loose probe—letting models complete truncated Wikipedia text—showed frontier models encode 95–98% of facts. A strict probe—four closed-book paraphrased questions, all must be correct—found 26–34% failure. The gap is partly a ruler artifact, but its shape holds: cold facts encode nearly as well as hot ones yet recall drops over 20 points. Thinking rescues 40–65% of encoded-but-missed facts vs. only 5–15% of never-encoded ones. The paper prescribes a triage ladder: rephrase, then multiple choice, then thinking, then retrieval—don't conflate empty shelves with lost keys.

Why it matters: Google Research used 2,150 Wikipedia facts across 13 models to separate facts that were never stored from facts that just weren't retrieved this time, measuring 95–98% encoding rates while closed-book failure rates still ran 26–34%.

Read the original ↗Export Markdown