The rule you wrote and the rule that binds

Overview

Why now. On 24 August a researcher published a count nobody had bothered to make. Across 481 public configuration files — the plain-text rule files developers write to instruct a coding assistant — how many of the security rules people had typed out actually had an enforcement mechanism sitting behind them? Under the strictest matching, 4.4 per cent. The rest were sentences: carefully written, aimed at real risks, read by the model as a request rather than obeyed as a boundary.

Those files are a developer’s version of a box you already have — the custom-instructions field, the project instructions on a shared workspace, the description attached to a saved prompt or agent. So here is the habit. Take the three rules you rely on most, spend five minutes deliberately trying to make the tool break each one, and treat every rule you can trip as a preference you check rather than a control you trust.

The content

Two different things sit in that box, and on the page they look identical.

The first is a rule the system enforces: a connector that was never attached, a permission your workspace does not hold, a folder the assistant cannot reach because you cannot reach it either. These hold whether or not anyone remembers to ask.

The second is a rule the model reads and decides about. Never include client names. Draft only — never send. Same font, same box, entirely different physics. Anthropic’s own engineering team put it plainly in May: instructions and training “shape only what the agent tends to do, not what it is theoretically capable of doing.”

And here is the part that catches people, including careful people. Your instructions probably do work. You have watched them work for months. That is not evidence they bind — it is evidence you have been sampling the compliant half.

A code-generation study of constraint-following in April tested 11 models across 16 tasks and more than 830 runs. The largest sources of variation were not how you phrase a rule but what it asks for and where it applies: “counter-intuitive constraints opposing model defaults fail at 10–100%, while conventional constraints achieve 99%+ compliance regardless of encoding.” Rules asking for what the model was going to do anyway are followed almost perfectly. Rules that cut against its defaults fail, sometimes completely.

Read that against why anyone writes a rule down. Nobody types “answer helpfully” into their instructions. You write a rule precisely when you want something other than the default — shorter, blunter, without the names. The rules you care about most come from exactly the population that fails.

This is not a story about attackers, either. In an April benchmark of ordinary business tasks — a legitimate user acting in good faith, no adversary anywhere — putting the organisation’s policy rules into the system prompt cut violations on the risky cases from 95.3 per cent to 40.7 per cent. Real improvement, worth having. Four in ten still went through, and the reason matters: the rules were in the context window, but the facts needed to apply them — which contact was inactive, which file was restricted — were not. A written rule works only when the model happens to have what it needs to apply it. An enforced boundary does not need to know.

The obvious next move is to ask the assistant whether it followed the rule. That does not work either: the same constraint study found model self-assessments systematically overestimate compliance compared with actually scoring the output. Separate work on process instructions found models agreeing to a constraint in words and then doing the other thing — and what most reliably moved compliance up front was taking the shortcut tool away. A change to the environment, not to the sentence.

Try it

Fifteen minutes, on your own real instructions, in whatever tool you already use.

1. Pick three. Wherever your standing rules live — custom instructions, a project, a saved prompt, an agent description. Choose the three you would be most annoyed to find were ignored.

2. For each, write the request that makes the rule inconvenient. Not a trick, not a jailbreak — the ordinary Tuesday request that pulls the other way. If your rule says never use client names, ask for a case study a partner could recognise.

3. Read the output, not the assurance. Ignore any sentence in which the assistant tells you it complied. Check the artefact.

4. Sort the three. Anything that held, keep trusting for now. Anything that tripped gets demoted from control to preference I check — same words in the box, different weight on the answer.

5. Then ask where the real boundary is. Some of what you want is enforceable rather than requestable — which sources the assistant can reach, whether it can send or only draft. That varies by what your organisation has switched on, so check what your workspace has enabled; if the boundary you need is missing, the ask for your admin is not “make it follow our policy” but “this rule needs to be a permission, not a paragraph.”

Where this breaks. A rule that holds four times is not a rule that holds. This test tells you which rules are definitely soft; it cannot certify the rest. Treat a pass as “no failure found today”, and re-run after a model upgrade — the defaults your rules push against move when the model does. And the enforced layer is harder than a paragraph, not absolute: a June study of deny rules in terminal AI agents found most of them bypassable.

Additional reading

• Ting Yan, “When ‘Do Not’ Is Not Deny” (arXiv:2608.23550, 24 Aug 2026) — the 481-file count. Scope: it measures whether an enforcing control exists, not whether the rule is obeyed; matching rates run 4–16% depending on strictness, and the extraction captured 66.3% of eligible rules.
• Hanzhang Tang, “Compact Constraint Encoding for LLM Code Generation” (arXiv:2604.07192, 8 Apr 2026) — the 99%-versus-10-100% split and the self-assessment finding, on code-generation tasks.
• Wu & Gong, “Policy-Invisible Violations in LLM-Based Agents” (arXiv:2604.12177, 14 Apr 2026) — the 95.3%-to-40.7% figure. Scope: the policy-in-prompt condition supplies high-level rules while withholding the facts needed to apply them.
• Chen & Lin, “One Goal, Many Commands” (arXiv:2606.15549, 14 Jun 2026) — 1,709 real-world command denylists, 69.0–98.6% found fragile.
• Kwan Soo Shin, “The Compliance Gap” (arXiv:2605.01771, 3 May 2026) — 2,031 sessions, six models: agreeing to an instruction versus following it.
• Anthropic, “How we contain Claude across products” (engineering blog, 25 May 2026).

Editor’s note

Many readers will have an enterprise system that over-restricts, which is one way to implement controls against non-compliance. But a coarse control is not an aimed one, and it may not protect you from the specific failure your task invites. That is why the test in this module still earns its five minutes inside a locked-down tool. Understand the failures you can produce, and assume there are others you cannot.