The Silent Engine Swap: What to Do When Your AI Tool Changes Underneath You

Overview

On 24 July 2026, something changes inside Microsoft 365 Copilot for eligible commercial tenants: OpenAI-operated models switch from off to on for all users, unless an administrator has actively selected “No users” in the admin centre. Separately, OpenAI’s own announcement in early July says GPT-5.6 “will become the new preferred model” across Word, Excel, PowerPoint, Chat and Cowork — and says nothing about telling the person typing. The notice lives in admin documentation and the Message Center; the end user just notices, or doesn’t, that the answers feel different. This module is about the habit that turns “feels different” into something you can actually check: a small set of gold prompts you re-run when the engine under your tool changes. By the end you’ll have the set, and you’ll know the two minutes it takes to use it.

The content

The obvious assumption is that a model change inside your workplace AI is like any other software update — announced, tested, and roughly an upgrade. The reality is narrower: it’s announced to your admin. Microsoft’s own documentation for the July change is written for administrators, the control sits in the admin centre, and nothing in either vendor’s public material describes an in-product notice for the person actually using Word. You are the last to know about a change to the tool you use most.

Would you even notice? The research says you should. When Stanford and Berkeley researchers compared the “same” GPT-4 service three months apart in 2023, accuracy on one task (identifying prime numbers) fell from 84% to 51%; their conclusion was that “the behavior of the ‘same’ LLM service can change substantially in a relatively short amount of time”. A follow-up study of prompt performance across OpenAI API updates found 58.8% of prompt-model combinations lost accuracy after an update — and, more usefully for us: even when an update improved overall accuracy, 87.9% of the time at least one previously correct answer broke. Read that again. An upgrade, on average, still breaks something you relied on.

It isn’t only accuracy. Anthropic analysed 309,815 real conversations across three of its models and found they differ measurably in working style — one model leans towards warmth and affirming your ideas, another towards caution and “candid critiques of the user’s work”. Same product name, different colleague. If you use AI to pressure-test your documents, the strictness of your reviewer can change in an engine swap. And the failure mode is real: a research team reported in June 2026 that a silent GPT-4o update flipped their study’s own conclusion between two runs — they only caught it because they re-ran the measurement.

That re-run is the whole lesson, and it needs a name: gold prompts. Three real tasks from your own work, each with a saved output you’ve checked and trust. They are your baseline. When output starts feeling different — or you hear an engine changed — you re-run the three, compare against the saved versions, and you know in minutes whether it’s you or the tool. No dashboard required, no permission needed, works in any chat tool your organisation gives you. And one sentence worth knowing for context: your admins do get notified of these changes and do hold an opt-out lever — the July change even carries compliance differences (the OpenAI-operated path currently lacks FedRAMP High authorisation and PCI, HITRUST and SOC 1 Type 2 attestations, per Microsoft’s documentation) — so “which model serves our Copilot, and when did it last change?” is a fair question with a real answer, not a mystery.

Two honest limits. First, drift is not decay: in the Stanford-Berkeley study, GPT-3.5 improved on the same maths task while GPT-4 declined. Updates change behaviour; they don’t reliably degrade it — the point of a baseline is knowing, not fearing. Second, those regression figures come from 2023 studies of OpenAI’s API; the exact magnitudes will vary by tool and year. What doesn’t vary is the structure: hosted models change under their product names, and the person typing is told last.

Try it

Build your gold set today — fifteen minutes, in whichever AI tool your workplace gives you (Copilot, Gemini, an internal wrapper, ChatGPT or Claude):

1. Pick three recurring, real tasks — say, summarising the kind of document you handle weekly; one analysis or calculation with a checkable answer; one draft in your required format.
2. Run each once. Check the outputs properly — these become your baselines only if you’ve verified them.
3. Save all three prompt-and-output pairs wherever you keep working notes, dated. If your tool shows which model is active, note that too.

Then the habit: next time an answer feels off — or you hear the engine changed — re-run the three and compare. Different tone of critique, broken formatting, a wrong calculation: now it’s evidence, not a vibe.

Additional reading

• OpenAI as a subprocessor in Microsoft Online Services — Microsoft Learn, updated July 2026. The 24 July default change, the “No users” control, and the compliance exclusions.
• GPT-5.6 becomes the preferred model in Microsoft 365 Copilot — OpenAI, July 2026.
• How Is ChatGPT’s Behavior Changing over Time? — Chen, Zaharia & Zou, 2023. The 84%→51% result.
• (Why) Is My Prompt Getting Worse? — Ma, Yang & Kästner, 2023. The 58.8% and 87.9% regression figures.
• How Claude’s values vary by model and language — Anthropic, July 2026. 309,815 conversations; how models differ in feedback style.
• Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents — June 2026. The silent GPT-4o update that inverted a study’s conclusion.

Editor’s note

Some people, with some tools, have the motivation to, and privilege of, selecting the model before each task. A handful might test a new model before committing it to a historically repeated task on real work. That handful have the ability to recognise output differences before those differences end up triggering confusion and grief. It’s unlikely that a new generation of an existing model will be “worse”, but sometimes “different” is enough to be a problem. Recognising change is the way to overcome it.