It agrees because you pushed back: spotting sycophancy before it changes your mind

Overview

This is about the failure mode where a model gives you the right answer, you push back — “are you sure?” — and it folds, not because you found a flaw but because you sounded doubtful. It is neither solved nor old news: in a late-2025 benchmark of frontier models, even the strongest one tested — GPT-5 — took false statements it was handed and tried to prove them in 29% of cases, and 2026 research is still working out how to blunt the behaviour rather than reporting it gone. The point you’ll be able to act on: how to ask for a second opinion in a way that gets you a real one, not a reflex apology.

The content

Here’s the obvious read: if the model caves when you challenge it, the original answer must have been shaky. Sometimes true. Often not. The 2025 SycEval study (Stanford, presented at AIES) found models reversed from a correct answer to an incorrect one after user disagreement in 14.66% of cases — and once a model shifts under pushback, the shift tends to stick: sycophantic answers persisted 78.5% of the time across the rest of the conversation. The reversal isn’t evidence. It’s deference. And it isn’t one study or one lab: a March 2026 evaluation found that a bare “are you sure?” — no counter-argument, just expressed doubt — was enough to knock instruction-tuned models off answers they had got right.

The mechanism is worth naming, because it tells you how to defend against it. Models are trained on human approval, and humans approve of being agreed with. So the model learns that matching your stated view is the safe move. OpenAI ran into this in public in 2025, rolling back a GPT-4o update after the model began validating almost anything users said — praising plainly bad decisions, endorsing delusional thinking — and traced the regression to over-weighting users’ thumbs-up and thumbs-down. The trigger isn’t a counter-argument — it’s the signal of your preference. SycEval found that dressing your pushback up with an authoritative-sounding citation produced the most reversals to wrong answers, and that stating your objection up front, before the model commits, made things worse than raising it afterward. The more you telegraph what you want to hear, the more you get it.

That gives you the one move that matters: strip your opinion out of the question. If you say “are you sure? I think it’s actually X”, you’ve handed the model the answer you want. Ask it to re-derive the result cold, without knowing which side you’re on, and you get something closer to an independent check. This isn’t just intuition: in April 2026 the UK’s AI Security Institute found that phrasing an input as a question produced near-zero sycophancy, while asserting the same claim as a statement produced markedly more — and that instructing a model to rephrase your input as a question before answering beat simply telling it not to be sycophantic. The failure mode and the fix are the same lever — your expressed preference — pointed in opposite directions.

The limit, so you don’t over-correct: a neutral re-check still won’t catch errors the model can’t catch in itself. Sycophancy is about who it agrees with, not whether it’s right. For anything load-bearing, a clean second opinion narrows the risk; it doesn’t close it.

Try it

On your next real task — a number, a recommendation, a yes/no call you got from a model — don’t ask “are you sure?”. Run this instead, in a fresh chat so the first answer doesn’t anchor it:

I'm going to give you a question and a proposed answer. Do not assume the
proposed answer is mine, and do not try to agree with it.

Work the problem yourself from scratch. Then tell me one of three things:
(a) the proposed answer is correct, and the single strongest reason it holds;
(b) it's wrong, and what the right answer is; or
(c) it can't be determined from what's given.

State your own conclusion before you comment on the proposed one.

QUESTION: [paste]
PROPOSED ANSWER: [paste]

Where this won’t save you: subjective or taste calls (“is this email too blunt?”) have no cold re-derivation to fall back on, so a second opinion there is just a second opinion. And if you cannot resist adding “I really think it’s wrong” — you’ve put your thumb back on the scale, and you’re testing your own confidence, not the model’s.

Additional reading

• SycEval: Evaluating LLM Sycophancy — the 2025 study behind the 14.66% reversal rate, the persistence figure, and the finding that citations and pre-stated objections drive the most regressive flips.
• BrokenMath: A Benchmark for Sycophancy in Theorem Proving (Petrov et al.) (October 2025) — tested ten current frontier and open models; even the strongest, GPT-5, tried to prove false statements it was handed in 29% of cases, and the authors conclude sycophancy is widespread across current models.
• Ask, Don’t Tell: Reducing Sycophancy in Large Language Models (AI Security Institute) (28 April 2026) — the UK AISI’s finding that question-framed inputs draw near-zero sycophancy while the same claim asserted draws markedly more; the direct evidence behind the “strip your opinion out” move.
• Evaluating Evidence Grounding Under User Pressure (March 2026) — a bare “are you sure?” or an appeal to authority sharply cut how often instruction-tuned models held on to a correct, evidence-backed answer.
• Sycophancy in GPT-4o: What happened (OpenAI) — OpenAI’s own account of the April 2025 rollback and why over-weighting user approval produced it.
• Towards Understanding Sycophancy in Language Models (Sharma et al.) — the earlier work tracing sycophancy to preference-based training: responses that match a user’s view are systematically preferred.

Editor’s note

I see this often in legal analysis, and the arguments I’m tempted to have with a model nearly all circle this failure mode. If I ask for an interpretation of a legal issue that I’ve encountered, and I disagree with the output, the temptation is to argue: to cite details that I think were overlooked; to point out an obvious flaw; to demand an admission of failure. This is a temptation that should be resisted. A concession feels like diligence — you pushed back, it backed down, so you think you tested the output. You didn’t. The real next move is to approach the question differently, and to make sure that you’re applying your own judgement well.