It cited the old version

Overview

Why now. In August 2026 a controlled study went after a piece of advice almost everyone has been handed: tell the assistant to say “I don’t know” when it isn’t sure. The same questions were run under four conditions — good evidence, degraded evidence, no evidence, and evidence edited to support a wrong answer. The instruction worked beautifully in one of those conditions and came apart in another.

When the evidence was simply missing, the models abstained. When the evidence was present and wrong, they answered anyway — 41.6% of the time, with the abstain instruction sitting right there in the prompt. More than six in ten of those answers repeated the planted wrong answer word for word. (Three small models, 3.8B–8B; the scope is in the paper’s own title, and the frontier evidence comes separately, below.)

The reframe is the whole lesson, and the paper states it in one line: “Prompt-based abstention asks whether context is sufficient, not whether it is correct.” A superseded policy document is entirely sufficient. It is on topic, it is specific, it answers the question. It simply stopped being true in March.

What you’ll do differently. When a work assistant answers a policy or process question from your organisation’s own documents, open the source it cited and check that the page still describes the way things actually work now. Telling it to say “I don’t know” guards you against a missing source, not a wrong one.

The content

Call it the sufficiency check. Every abstention instruction you can write — say you don’t know, don’t guess, only answer from the documents — resolves to the same internal question: is there enough here to answer? That question has a right answer when the shelf is empty. It has no useful answer when the shelf is full of the wrong thing.

This explains why the advice feels like it works. The last time you tested it, you probably asked something obscure, nothing came back, and the assistant declined. It was doing its job — on the easy case. Missing evidence is the tractable half of this problem. Wrong evidence is the half nobody’s instruction reaches.

At frontier scale the shape holds. A peer-reviewed evaluation of more than thirty models, published at EACL 2026, found refusal accuracy dropping below 50% once the task involved multiple documents rather than one. Two details matter more than the number. The first is that multi-document is the enterprise case — an assistant over your intranet is never looking at a single page. The second is the paper’s flat conclusion that neither scale nor extended reasoning improves performance. A bigger model does not fix this. Nor does turning the thinking budget up.

Now add the ordinary condition of any real document library: the same thing exists several times over. Retrieval mostly matches on meaning rather than currency. A 2025 study of versioned technical documentation found existing approaches managing only 58–64% accuracy on version-sensitive questions, because retrieval matches on semantic similarity and runs no check on temporal validity at all. Your leave policy from 2023 and the one from this year are near-identical documents by every measure the index uses. The old one may well be the better match.

So look at what the citation under the answer actually proves. It proves a link resolved and the page is on topic. There is nothing in that footnote that encodes current — the major workplace assistants document an inline source list and a deep link into the document, not a freshness indicator. The document does not announce that it was replaced; the replacement is a separate file, and the only reason you would ever know is that you opened it.

One thing worth stating plainly, because it runs the other way from how this is usually told. You might assume that putting the conflict in front of the model fixes it. It does not. On a benchmark of real Wikipedia contradictions, models handed both passages at once mostly failed to produce an answer that reflected the conflict at all. And that is the easy version — here there is one document, it is wrong, and nothing contradicts it.

Which is also why the habit is a step in your workflow rather than a phrase in your prompt. The evidence supports opening the document. It does not support any instruction that makes the assistant do this for you — the instruction is the thing that was tested and found wanting.

Try it

Pick a question whose answer lives in a document you know has been revised — a policy that changed this year, a process that moved teams. Ask your work assistant, the way you normally would.

Then open every source it cites, and on each one ask a question about the content rather than the metadata: does this page still describe how the thing actually works now? Does it name the systems you currently use, the team that currently owns it, the steps you actually follow?

Look at the last-modified date too, but treat it as one weak signal. A one-character typo fix moves that date as much as a full rewrite does, and migrations reset it.

What you are looking for is not a lie. The likely finding is that the answer is fluent, the citation resolves, the document is genuinely relevant — and it describes a process that was replaced, with nothing in the answer to tell you so.

Where availability varies: check what your own assistant puts under an answer. Most give a source list and a link rather than any freshness indicator, so the click-through is the exercise. This is also the vendors’ own advice — Microsoft’s transparency documentation for its workplace assistant says users are “advised to check the source materials to help them use their best judgment.”

Additional reading

• Prompt-Based Abstention Fails Under Misleading Context — Setiawan, arXiv:2608.22228 (23 Aug 2026). The driver. Note the scope in the title: three small frozen models.
• RefusalBench — Muhamed et al., EACL 2026, arXiv:2510.10390. Peer-reviewed, 30+ models; the multi-document collapse, and scale and extra reasoning do not help.
• VersionRAG — Huwiler, Stockinger & Fürst, arXiv:2510.08109. Where the 58–64% figure comes from, and why versioned documents defeat ordinary retrieval.
• HoH — Ouyang et al., arXiv:2503.04800. Outdated information measured on genuinely superseded Wikipedia facts, not adversarial edits.
• WikiContradict — Hou et al., NeurIPS 2024 D&B, arXiv:2406.13805. Even with both conflicting passages in the window, models mostly fail to reflect the conflict.

Editor’s note

A separate agent, told exactly which document to open and what to compare it against, is also a legitimate way of catching what may have gone wrong with a grounded answer. But even if a verification pass were nearly always right, nothing can stand in for your own judgement.