Convene a Panel: Using Multiple AI Personas to Stress-Test a Decision

Overview

This is a workflow for pressure-testing a decision: you ask one model to argue your plan from several conflicting viewpoints — the sceptic, the customer, the CFO, the devil’s advocate — and you read the disagreement between them. In December 2025, Wharton’s Generative AI Labs published a study finding that telling a model “you are a world-class expert” does not reliably improve whether it gets facts right; across six models and thousands of trials, the expert label mostly made no difference, and a mismatched one sometimes made things worse. In their practitioner write-up, the same team noted where personas do earn their keep — they change “what risks it flags and what opportunities it highlights.” By the end of this, you’ll know how to run a one-prompt review panel, and exactly when not to trust it.

The content

Here’s the obvious read: if you want a better answer, give the model an impressive job title. The Wharton work overturns that for anything with a right answer. Persona prompts shape style, not substance — useless for accuracy, occasionally counterproductive when the role doesn’t match the question.

But read the finding the other way and it hands you the use case. Personas change which considerations get surfaced. A compliance officer flags different things than a sales lead. That is worthless when you need a fact and decisive when you need a judgement — because most real decisions fail not on a wrong fact but on a viewpoint nobody held in the room.

So convene a panel. Instead of asking one neutral assistant “is this a good plan”, you instantiate several roles and make them disagree on purpose. The mechanism that does the work is structured disagreement: separate the seats, force each to argue from its own incentives, and the gaps between them are your blind spots made visible. This echoes the multi-agent debate research (Du et al., ICML 2024), where models that critique each other across rounds reason better than one model answering alone — the value is in the friction, not the consensus.

Where it breaks: the panel will sound confident and balanced even when it’s wrong on the underlying facts, because every persona draws on the same model. It cannot tell you whether your revenue figure is real — only whether your plan survives contact with the CFO who reads it. Use it for “what am I missing”, never for “is this number correct”. And if every persona agrees, that is not validation; it usually means your prompt was too thin to generate real conflict.

Try it

Take a real decision you’re weighing this week — a plan, a proposal, a hire, a pricing change. Paste it in, then run:

Here is a decision I'm about to make:
[paste your plan, proposal, or recommendation — include the reasoning, not just the conclusion]

Convene a panel of four people to stress-test it. Each must argue ONLY from their
own incentives and stay in role:
- A sceptical operator who has to deliver this and has seen plans like it fail
- The customer/end-user it affects, who didn't ask for it
- A CFO focused on cost, risk, and what we're not measuring
- A devil's advocate whose only job is to make the strongest case against

For each: give their single sharpest objection and the one question they'd ask me.
Then list the 3 disagreements BETWEEN the panellists — where they'd argue with
each other — because that's where my blind spots are.
End with the 2 weakest points in my plan that more than one persona attacked.
Do NOT try to reach consensus or reassure me.

The payoff is the cross-examination and the points multiple seats attacked — not a tidy verdict. Then do the verification the panel can’t: check the facts and figures yourself.

Additional reading

• Playing Pretend: Expert Personas Don’t Improve Factual Accuracy — Wharton Generative AI Labs (7 December 2025). The study behind the “shape, not substance” reframe: six models, thousands of trials, no reliable accuracy gain from expert personas.
• Stop Telling AI Who to Be: Why Personas Don’t Improve Accuracy — Wharton AI & Analytics Initiative (26 March 2026). The practitioner summary, including where personas do add value: tone, framing, and which risks get flagged.
• Improving Factuality and Reasoning in Language Models through Multiagent Debate — Du, Li, Torralba, Tenenbaum & Mordatch, ICML 2024. The evidence that structured disagreement between model instances sharpens reasoning.

Editor’s note

I run this workflow All. The. Time. It can be token-expensive, especially when you orchestrate it using evaluator feedback loops, but the varied opinions are worth it. One failure mode the module doesn’t cover comes from imperfect persona shaping. I once ran a board-of-advisors stress test on a design proposal for a challenging piece of work, and I deliberately included a devil’s advocate as one of my evaluator agents. The board determined that the devil’s advocate was correct, and that I shouldn’t do the work — not because the idea or the proposal was bad, but because the work itself was deemed too difficult. Evaluating the evaluation is critical; done well, it sharpens your own judgement.