It worked when you watched it: what a single success actually buys

Overview

You tried it once. It worked. So you saved the workflow, pointed it at the recurring job, and stopped watching.

That moment is what this module is about — because a benchmark published on 20 August 2026 lets you put a number on what that one successful run actually told you, and the number is smaller than it feels.

Here is the habit. Before you let a workflow run without you, run it on the same input three or four times and compare the results. And if it is the kind of job you cannot safely repeat, that is the one to keep watching rather than the one to trust.

Call it the repeat test. It costs a few minutes, and it is the whole distance between I saw it work and it works.

The content

The finding. Thinkingbox-bench is a set of 507 business workflows — retail, hospitality, auto insurance, internal IT and HR support — the kind of multi-step job where the assistant has to chase missing details, follow a policy, and leave the system in the right state at the end. Every task was run twenty times. The strongest model tested finished 65.36% of tasks on a single attempt. It succeeded on all twenty attempts on only 25.25%.

Now do the division, because that is the sentence worth carrying. If you watched a task succeed once, the chance it belongs to the set that works every time is 25.25 divided by 65.36 — about 38.6%. One success buys you roughly two-in-five odds.

That is a better claim than “one success means nothing”, which is false. It is also a long way from what most of us act on.

Why repetition is the right instrument. If each run were an independent roll of the dice at those odds, twenty-for-twenty would essentially never happen. It happened on a quarter of the tasks. So success is not spread evenly: some workflows are genuinely solid, and the rest are genuinely erratic. That is precisely the distinction repetition can draw — and why three or four runs are worth so much more than one.

This is not one surprising paper. Two years and several model generations earlier, τ-bench reported the same collapse: agents managing fewer than half the tasks were, in its authors’ phrase, “quite inconsistent” across repeated trials. The measure used above — succeed on all k attempts, not just one — comes from that work. What is new is a figure you can divide yourself.

The failure you will not notice. The uncomfortable half of the benchmark is what the failures looked like: “many failed trials show clean termination and valid state-changing actions”. The run ended tidily. Real actions were taken. Records changed. It simply did not do the job. Which means the completion signal — the confident closing summary, the tick, the word done — is not the thing to check. The end state is.

The asymmetry, which is the actual problem. Re-running is nearly free when nothing changes: summarise this inbox, draft this brief, pull these figures. Most saved workflows a knowledge worker really has are that kind, and the repeat test costs you a coffee. But it is impossible for the ones that alter something real. You cannot file the same expense claim twenty times to find out whether filing works. So the workflows where reliability matters most are exactly the ones repetition cannot test. For those the posture has to do the whole job: keep a hand on it, and check what changed rather than what it said. Keep some of it for the repeatable ones too — a good test tells you what a workflow tends to do, not what it did this morning.

The hopeful half, carefully stated. A re-analysis of nearly ten thousand agent runs found tidy-looking failure is far rarer when a second party can independently check the state. Where only the agent could write to that state, false success was 45% and 48% of failures; where the other side could also act and verify, 3%. The authors call that an observation rather than a cause — and it does not mean fewer failures. It means fewer failures wearing the costume of success, which is exactly what checking the end state yourself buys you.

And the limits. The benchmark is days old and not peer-reviewed, and its simulated user shares a model family with the top-scoring agent — which the authors flag themselves. The false-success work is a single-author workshop re-analysis. None of that moves the habit, which costs little and is right regardless.

Try it

Do this today, on a workflow you already rely on.

1. Pick a read-only one. A saved prompt, agent or chain that summarises, drafts, extracts or looks something up — nothing that sends, files or updates.
2. Run it four times on exactly the same input. Same document, same question, same wording. Fresh conversation each time.
3. Compare on substance, not phrasing. Wording will always vary; that is not the signal. Look at whether the same figures appear, whether the same items are found, whether a step is quietly skipped in one run and not the others.
4. Read the divergence as the finding. If the four agree, you have earned considerably more than the 38.6% a single run gave you. If they disagree, you have just found the fault on your own time rather than in front of someone else.
5. For the jobs you cannot repeat, write down the end state you expect before you run it — the record that should exist, the field that should change — then go and check that directly, rather than reading the summary of what happened.

Some agent-building tools offer a trial counter and an automatic comparison across runs, but usually at the maker tier rather than the everyday user tier — worth checking what your workspace has enabled, and worth asking for if it hasn’t. Otherwise, do it by hand. Four runs and a careful read is the entire method.

Additional reading

• One Success Isn’t Reliability: Thinkingbox — the 507-workflow benchmark, the twenty-trial design, and the pass@1 versus pass^20 gap. Note the authors’ own limitation on the shared model family between simulator and top agent.
• τ-bench — where the “succeed on every attempt” metric comes from, and the same inconsistency two years earlier.
• From Confident Closing to Silent Failure — the characterisation of false success, and the finding that judges on τ²-bench lean on confident closing language rather than verified state changes.

Editor’s note

Being blunt: I’m not sure that there’s ever a time when an automated workflow should be exempt from quality control review by you. I review all of my automated workflow output regularly. Some of them daily. There are some that I iterate on frequently, and others that haven’t given me any issues in months. The quality control step is a part of working with AI. This module is mainly a reminder of that. Test it before you turn it loose on real work, and iterate when you notice that something isn’t right.