Break your own agent: designing for failure at age eleven
The most useful question a child can ask about their own agent is how it could go wrong. Here is the exercise we run to make eleven-year-olds design for failure before they design for success.
Before we let any child's agent run for real, we ask one question that reliably produces the most useful ten minutes of the whole session: how could this go wrong? An eleven-year-old designing a homework-reminder agent came up with three answers in under a minute - it could nag about an assignment that was already submitted, it could message the wrong sibling's device, and it could keep reminding about something that got cancelled. None of those had occurred to him while he was designing the happy path.
That gap - between designing what should happen and designing for what could go wrong - is exactly what this article is about, and it turns out to be a skill children pick up faster than most adults expect.
How do you stop an AI agent going wrong?
You do not stop every possible failure. You find the likely ones ahead of time, on paper, before the agent runs unattended, and you attach one rule to each one. This is cheaper and more effective than watching for problems after they've already happened.
Why naming the failure has to come first
"It might go wrong somehow" produces no useful rule, because there is nothing specific to design against. "It might remind me about a cancelled assignment" produces an obvious fix immediately - check whether the assignment is still active before sending a reminder. Specificity is what turns a vague worry into an actual safeguard.
Designing the failure is part of designing the thing. It is not a separate, optional step you get to if there's time.
This gets easier with practice, not harder
The first time we run this exercise, most children can only think of one failure, and it's usually vague. By the third or fourth agent they design, they start finding three or four specific ones without being asked twice - not because the agents got more complicated, but because they've learned what a real failure mode actually looks like, rather than a generic worry.
The honest limit is that no amount of imagining failures in advance catches every one that will actually happen - new situations produce new mistakes nobody thought to name. That is exactly why the approval gates from the previous article still matter even after this exercise is done well; imagined failures get a rule in advance, unimagined ones still need a human checking in at the right point. This closes out agents. The next level, products, is where a child stops designing how a tool behaves and starts designing something other people actually use.
Questions we get asked
How do you stop an AI agent from going wrong?
Before letting it run unattended, deliberately list specific, named ways it could fail - not a vague worry, but a concrete mistake like sending a reminder for a cancelled task - and attach one rule or check to each one. This catches problems on paper, before they happen for real, rather than only after.
Why should children design for failure before building an AI agent?
Because imagining specific failure modes in advance is far cheaper than discovering them after an agent has already acted on its own. It also teaches a transferable habit: naming exactly what could go wrong produces a usable fix, while a vague worry about something going wrong somehow does not.
Does planning for failure mean an AI agent will never make mistakes?
No - planned-for failures get a rule in advance, but new, unimagined situations can still produce mistakes nobody thought to check for. This is exactly why approval gates at the highest-stakes steps remain useful even after a careful failure-planning exercise; the two safeguards cover different gaps.