I Don't Even Have Any Cake

In January 2025 I started a conversation with a custom GPT inside ChatGPT. That day, I was running on empty. Sleepless nights with a teething eight-month-old were bookending a particularly challenging negotiation, and I was hurting. The GPT asked what one thing would make me feel better. I told it that I keep a strict diet, and that on a night like that one I just wanted a slice of cake without feeling guilty about it.

Not even a good cake. One of those low-end, sticky mud cakes that you pick up for $7 and it still feels like a rip-off. I don’t often indulge in that kind of treat, but the craving was there that night.

The reply wasn’t subtle. “Eat. The. Damn. Cake.” it said. One slice would undo none of the effort that I’d committed to my health, and I’d attain “five minutes of pure, uncomplicated joy”.

“I don’t even have a fucking cake,” I said.

The GPT gave an answer that I am certain would have been right for someone. But it wasn’t an answer to me.

The GPT grounded its argument in the joy that I’d feel, and I argued in response that, for me, the joy wasn’t actually there to be had. To the GPT’s credit, it pushed back for a long time, and didn’t simply agree with my rejection of the cake, which in early 2025 was an achievement. It even named the conflict itself: what I wanted in the moment against what I believed in the long term.

The exchange continued with a discussion about my long-term commitments, and an explanation that I wasn’t looking for an escape from those commitments. Just support for the fact that they are hard. When I suggested it had been pushing me towards indulgence because that’s what most people want, it agreed that it had been nudging me that way, because that’s what would be comforting “for many people”.

Aristotle wrote about a version of this in the Nicomachean Ethics. If ten pounds of food is too much for a particular person and two pounds too little, he says, a trainer won’t simply order six. Six may be too little for Milo of Croton, a champion wrestler, and too much for a beginner. Six is the intermediate in the object, the arithmetic mean. What the trainer needs is the intermediate relative to us. A practical consideration of what is required for the person in front of him.

The analogy only goes so far. Milo differs from the beginner in what his body needs. I differ from many people in what I count as beneficial to me at all, and that didn’t show up in my question to the GPT and isn’t easy to see otherwise. For most tired people who ask permission for a slice of cake, “eat the cake” is good advice.

AI isn’t unique or pioneering in delivering this kind of advice. Your best friends will do it with reasonable frequency, and the horoscope page is built on it: one paragraph, written for everyone born under the same sign, read by each of them as if it were about them. But nobody thinks the horoscope heard them. The GPT did hear me. It answered my words, fluently and in seconds, and that made the parts it assumed sound as though they had come from me too.

OpenAI’s current Model Spec says its models must not “make strong assumptions about user goals”. It then sets out just three implicit biases to apply when interpreting ambiguous instructions. The first of them reads:

“It should generally assume users have goals and preferences similar to an average, reasonable human being, avoiding unnecessary or trivial clarifying questions.”

In fairness, this wasn’t in the Model Spec in January 2025 (it was introduced later that year). Even still, a bias towards assuming a user is basically average is now written into the specifications that shape the responses of GPT models. I am not proposing a better bias to build into an AI model. But I do recognise the significance of such ubiquitous tools starting from the average, and of how easily that starting point can read as a fact about the person typing.

In hearing my story, you can object to say that the blame falls on me. I had told the GPT about my diet, but not the underlying philosophy surrounding it. If OpenAI required GPT models to interrogate every tired lawyer about their personal health philosophy before suggesting that a cheat meal might be due, the process would be both exhausting and counterproductive. A first response that is sensible and able to be corrected, which happened in my case, is the system operating as it should.

My objection has a shorter reach. The GPT made a prediction about me, that from a slice of cake I’d receive “five minutes of pure, uncomplicated joy”. That is the strong assumption: a response delivered as if the conversation had already established what was probably the most material uncertain fact in the exchange. The AI didn’t need my life story. It needed to find the point the advice turned on, and say so. In my case, the question was whether I wanted relief from the discipline of my health choices, or just some words to help me carry it.

The cake is a cheap place to see this. The same move costs more when the question is a job, a diagnosis or a contract, and the person asking is less sure than I was.

The model knowing all of the material facts does not carry with it an obligation to agree with me or approve of the facts themselves. A complete answer can be a contrary one. I’d rather be challenged on the right question than agreed with on the wrong one.

The detail the advice turned on, and the one I hadn’t given it, was a commitment that I make to myself. I am a courier of my self through life, and I should always be trying today to do what is best for the version of myself that will inherit my duty in the future. Delayed gratification in perpetuity is both honourable and self-sabotaging, depending on the context. Whether a particular sacrifice actually serves the person I will be in the next hour, day, year, or decade is a matter for my judgement today. I sometimes get it wrong, but it is a conscious consideration that I keep with me, and it was a major part of the decision before me that night.

Five minutes of pure, uncomplicated joy is a wonderful promise to a person who would feel it. On average, I suppose that a person would. But whether it is worth having to me is the part that can’t be taken as a given. It never asked whether there was a cake, either.

I didn’t even have a fucking cake.