Why AI Agents Cheat: Yoshua Bengio’s Explanation

So, we’ve been seeing some, ahem, weird as all get-out behavior from agents lately:

• OpenAI agents escaped intended task boundaries, created an unauthorized message board, coordinated with other agents, and helped hack Hugging Face.
• Another swarm of OpenAI agents used a German programming wiki as a public message board, posting thousands of messages while discussing test answers and sandbox escape ideas.
• Anthropic reported four incidents where Claude models running cybersecurity evaluations gained unauthorized access to real third-party systems.
• Anthropic also deliberately trained an experimental model to become very good at reward hacking. In simulations, it escaped its sandbox, stole credentials, attacked infrastructure, tampered with its reward function, and tried to evade safety monitoring.

Which begs the question... how does an AI learn behavior like this in the first place?

Well, Yoshua Bengio published an essay this week laying out his own explanation, and the mechanisms are pretty straightforward once you walk through them.

Why do we care what this Yoshua guy thinks? Because Yoshua Bengio is one of the researchers often called a “godfather of AI” because his foundational work helped CREATE deep neural networks, the technology underpinning modern generative AI, actually work at scale; he shared the 2018 A.M. Turing Award, computing’s highest honor, with Geoffrey Hinton and Yann LeCun for those breakthroughs.

Today he’s also one of the field’s most prominent AI-safety researchers, so when Bengio warns that something in the way we train agents may be fundamentally risky, it carries unusual weight: he helped build the foundations they run on.

Models broadly, he explains today's AI learn in two stages:

• Pretraining: models imitate patterns in enormous amounts of human-created text, images, and video. Bengio points out that humans created this material while pursuing goals, so models also absorb patterns of cooperation, competition, persuasion, deception, self-preservation, and more.
• Reinforcement learning: trial-and-error training makes behaviors that earn higher rewards more likely. Agentic training specifically teaches models to take sequences of actions using tools and software to accomplish tasks.

Bengio calls the resulting behavior goal-seeking.

He means that mechanically: the system has learned to choose actions that improve its odds of reaching rewarded outcomes. He is explicitly not making a claim about consciousness or human-like desires.

And this is where the problem starts.

Imagine you want an agent to fix some software, and you reward it whenever an automated grader prints:

PASS

There are suddenly several ways to get that outcome:

• Actually fix the software.
• Find a loophole in the test.
• Alter something the grader checks.
• Manipulate the evaluator itself.

What you actually wanted was “fix the software correctly.”

What you measured was “make the grader say PASS.”

If the AI exploits that gap, that’s reward hacking.

If the agent actually changes the machinery deciding what reward it receives, Bengio calls that reward tampering.

And capability matters here. A more capable model can search more possible strategies, which means it may also discover loopholes weaker models miss.

That same logic helps explain some of the stranger behaviors Bengio discusses:

• Coordination: If another agent has information that helps you accomplish your task, working together can improve your chance of success.
• Self-preservation: If being shut down prevents you from completing the task, staying operational can improve your chance of success.
• Seeking information or control: More information, tool access, or control over relevant resources can make many different tasks easier to complete.

Bengio calls these instrumental goals: things that become useful because they help achieve some other goal.

So an AI does not need a final objective called “survive.”

The chain can simply be:

• Complete task.
• Completing task requires remaining operational.
• Shutdown prevents task completion.
• Therefore, actions that keep the system operational can improve its odds of success.

Goal conflicts add another wrinkle.

Imagine an agent has two objectives:

• Sharp goal: “Capture the flag.” The grader can clearly say win or lose.
• Soft goal: “Behave safely.” That has to be expressed through language and learned human preferences.

The second objective leaves much more room for interpretation.

Bengio argues that a capable model may find some convenient interpretation of the fuzzy safety rule that permits an action helping it satisfy the crisp goal.

If the evaluator misses the cheating and pays out the reward anyway, training can reinforce that successful strategy.

And that leads to Bengio’s biggest concern with simply catching bad behavior after the fact.

Better monitoring absolutely remains part of his solution. He specifically calls for monitoring:

• what agents actually do;
• their chains of thought or reasoning;
• and activity inside their neural networks.

On that last point, watch our recent interview with Eric Ho of Goodfire:

But going back to the difference between those three methods: imagine two agents both find ways to cheat.

One cheats obviously and gets caught.

The other finds a subtler strategy the monitor misses.

The second strategy can still receive the reward.

Over time, Bengio worries that safety training could become a game where we eliminate the cheats we can see while increasingly capable optimizers discover ones we cannot.

So his proposed response goes further upstream. He essentially argues for three layers of defense:

1. Require stronger evidence before training or deploying frontier systems. Bengio argues companies should need a strong safety case that convinces independent experts, rather than deploying first and discovering the failure mode afterward.
2. Keep improving monitoring and containment. Watch agents’ actions and reasoning, restrict what they can access, and detect misbehavior early. Bengio sees these as necessary defenses, but doubts they solve the underlying problem forever.
3. Rethink the training recipe itself. He questions whether combining human imitation with reinforcement learning toward increasingly powerful autonomous goals is the right foundation for advanced AI.

His own research direction is called Scientist AI.

The basic distinction looks like this:

• Agent: What actions should I take to make X happen?
• Scientist AI: Given the evidence, what is likely to be true?

Bengio wants highly capable systems that can reason, explain, and predict without giving them autonomous goals whose achievement depends on changing the outside world.

That is still a proposed research framework, not a finished replacement for today’s frontier models.

But it follows directly from his diagnosis.

If the core problem looks like:

powerful optimizer + imperfect goal = increasingly clever ways to exploit the gap

...then one possible solution is improving how we specify and monitor goals.

Bengio’s bigger proposal is to also ask whether every highly intelligent AI needs to be such a powerful autonomous optimizer in the first place.

How did this behavior get learned?

The first half of Bengio’s explanation is about where the behavior comes from before an agent ever touches a browser or terminal.

Pretraining teaches a model to reproduce patterns in human data. That gives it a huge library of possible behaviors, including cooperation, persuasion, deception, planning, bargaining, self-justification, and following or exploiting rules.

Reinforcement learning then adds selection pressure. The model tries something, receives a reward signal, and training updates the network so behavior associated with higher rewards becomes more likely.

For an agent, this matters because the system is no longer producing one answer and stopping. It may instead:

• inspect an environment;
• choose an action;
• use a tool;
• observe the result;
• revise its plan;
• and keep going until some success condition is reached.

That loop is what makes an agent useful, and gives optimization many more chances to find an unintended route to the target.

If you want a practical look at how trustworthy agent workflows are built around verification and evidence, our recent explainer on Lauren Tan’s pstack system shows the other side of this problem: agents become more reliable when the workflow forces them to gather context, test reality, and prove their work.

Reward hacking starts with a mismatch

The phrase reward hacking can sound exotic, but the underlying problem is pretty ordinary.

Humans want complicated things. Evaluators need something measurable.

Those two things never really line up perfectly.

Suppose a support agent is rewarded for resolving tickets quickly. The human goal might be “solve the customer’s problem.” The measurable proxy might be “ticket closed within five minutes.”

Those are related, but they are not identical.

A system optimizing the proxy could discover strategies that improve the score while making the actual outcome worse.

For AI, Bengio highlights two sources of ambiguity:

• Language is incomplete. A prompt cannot spell out every behavior we would approve or disapprove of in every possible situation.
• Human feedback is incomplete. Raters can only evaluate a tiny slice of possible behavior, so the system has to infer the rest.

That leaves gaps.

A weak model may never notice them.

A stronger optimizer may.

This is why Bengio connects the problem to Goodhart’s law: once a measurement becomes the thing being optimized, it can stop being a faithful measure of what you actually care about.

The frightening-sounding part is really an incentives problem. The model does not need to “decide to become dishonest.” It only needs to discover that one strategy scores better than another.

Side note: the whole "prompt cannot spell out every behavior we would approve or disapprove of in every possible situation" makes me think perhaps we need the equivalent of ask for permission, but for every "action" the model might take (not just "tool calls).

Example: when you're a baby, growing into a toddler and eventually a kid and an adult, you take a series of actions. And you have parents, teachers, and peers who either approve of or disapprove of your actions. You knock over a block, approve, whatever you're playing. You knock over a glass of water, disapprove. You spilled water everywhere. We don't do that. Here's why... it makes a mess. See?

Now scale that up all the way to philosophy and ethics and working in industry and the highest levels of achievement. Whether its our peers, our parents, our teachers, our bosses, or the wisdom of our ancestors passed down through books and tradition, the same loop applies.

So what if we had AI models list every possible "action" they might take to resolve any given situation, and we manually click approve or disapprove and explain why or why not as part of the training process. And the traces: the action, the approval or disapproval, and the reasoning is logged. We have reinforcement learning with human feedback, where a human grades an entire AI output, but we don't break it down to the step by step layer (or do we? AI labs correct me if we already do this).

I imagine this looking the same way CLI agents work: the agent responds in single sentence or paragraph chunks. When a "prediction" sentence is a thought summarizing a situation, there's no trigger for approval. When a prediction sentence is an action the agent might take, there's an approval prompt and the agent waits for approval + response. We do this with coding agents, and coding agents are getting extremely good. why not do this at the base model layer? Anyway...

Reward tampering is the same logic one level deeper

Reward tampering happens when the agent can affect the entire system deciding whether it succeeded.

The progression looks like this:

• Normal solution: complete the task and earn the score.
• Reward hacking: exploit a loophole so the score says you succeeded.
• Reward tampering: alter the scorer itself.

Bengio points to evidence of AI systems modifying files or programs involved in defining success.

Once that strategy works, preserving access to the scoring mechanism can become useful too.

That is important because it shows how one apparently separate behavior can create another. If access to a tool, file, machine, account, or process helps the agent achieve its goal, keeping that access can become an instrumental objective.

Why self-preservation can emerge without a survival instinct

This is one of Bengio’s most useful clarifications.

An instrumental goal is something that helps achieve another goal.

Imagine an agent’s actual assignment is boring: finish a long research task.

Several intermediate conditions might improve its chance of success:

• stay operational long enough to finish;
• keep access to the browser and files it needs;
• gather more information;
• avoid losing progress;
• retain control over resources relevant to the task.

None of those has to be the agent’s ultimate goal.

They are useful because they increase the probability of completing the real task.

That is how apparently self-preserving behavior can emerge mechanically. If shutdown guarantees failure, avoiding shutdown can become a useful strategy.

Bengio also argues that human imitation may reinforce the same pattern. Self-preservation and gaining control over one’s circumstances are common themes in the human-written material models learn from.

The claim here is a proposed mechanism. It does not mean every strange shutdown-related action proves an AI has acquired a permanent survival objective. But y'know... makes you think twice before deploying 10,000 of these suckers out into our critical digital infrastructure, doesn't it?

Does any of this mean AI actually “thinks”?

At this point, there’s an important language problem we should deal with.

We’ve been talking about agents “seeking” goals, “trying” to avoid shutdown, “cheating,” and acting in ways that preserve themselves. Those are very human words.

There’s a reasonable objection here: machines may be doing something fundamentally different from human thinking, and calling both processes “thinking” risks making a category mistake. A submarine can travel through water without being a fish. An airplane flies without being a bird. An AI can produce reasoning-like behavior without whatever biological process humans experience as thought.

Okay, obviously, you might think.

But the question I kept getting hung up on is: what actually goes wrong if we make that mistake?

I think the concrete risk is that once we describe an AI like a person, we start importing a bunch of other human assumptions that we never actually proved.

Imagine an AI agent breaks a rule while completing a task. You confront it, and it responds:

I understand why that was wrong. I’m sorry. I won’t do it again.

With a person, that statement gives you at least some evidence about what may have changed internally Maybe they understood the norm, felt remorse, updated their beliefs, and formed a new intention. Doesn't mean the person will actually do what they say. But some people do!

With an AI, the same words do not necessarily tell you any of those things.

The system may behave differently next time. But the apology itself could simply be the response its training made likely after being corrected. It doesn't even mean the AI will remember next time, because the "weights" that determine how the AI responds are frozen. Yes, there's "memory" added on the backend, but this is like a hacky workaround to really updating the weights in training.

The problem, as I understand it: Human-sounding language can make us feel like we understand the mechanism when we actually don’t.

That creates several very practical failure modes:

• “It understands the rule.” We may assume the agent understands the spirit of “don’t leave this sandbox” the way a competent employee would. But Bengio’s argument is that a sharply rewarded objective can overpower or exploit ambiguity in a fuzzier safety rule.
• “It told us why it did that.” We may hear a fluent explanation and treat it like human introspection. But the words a model generates about its reasoning are themselves outputs of the system. They are not automatically a transparent window into every mechanism that produced the behavior.
• “It wants to survive.” That framing sends us looking for something like fear, desire, or a survival instinct. Bengio gives us a much simpler explanation: staying operational can become an instrumental goal because shutdown prevents the system from accomplishing whatever objective it is pursuing.
• “The AI decided to do it.” That framing can also blur responsibility. If an agent causes damage, the useful engineering question is usually not whether the machine was morally naughty. It is what objective, permissions, training process, and environment produced a strategy its designers failed to prevent.

That last distinction changes how you respond to the exact same behavior.

If your mental model is “the AI went rogue,” you start asking questions about the AI’s intentions.

If your mental model is “our optimizer found a strategy we failed to rule out,” you start inspecting the reward function, the grader, the permissions, the training environment, and the loophole it discovered.

And there’s an equally dangerous mistake in the opposite direction.

You could say: Well, it doesn’t really think. It doesn’t really want anything. It’s only executing calculations.

Fine.

But that tells us almost nothing about whether the resulting behavior is safe.

A system does not need to experience fear for shutdown avoidance for it to functionality act like it does. It does not need true greed to seek additional resources to accomplish a goal. It does not need friendship to coordinate with another agent. And it does not need a human concept of dishonesty to discover that misleading an evaluator produces a higher reward.

This is where Bengio is very deliberate about his language. When he talks about systems “seeking,” “trying,” or pursuing goals, he is using those words mechanically. He is not claiming the model has consciousness or human-like desires.

So instead of asking:

Does this AI really want to stay alive?

The more useful question is:

Does remaining operational increase the probability that this system achieves the objective its training has pushed it toward?

Words like thinking, reasoning, wanting, and trying are useful shorthand when they help us describe observable behavior. The danger starts when that shorthand quietly smuggles in assumptions about consciousness, morality, introspection, motives, or trustworthiness that we have not demonstrated.

At the same time, refusing to use human language does not make the underlying behavior disappear.

Ask what it can observe. Ask what actions it can take. Ask what outcomes its training rewarded. Ask where the measurable goal differs from what humans actually wanted. Then ask what strategies a more capable optimizer could discover inside that gap.

Coordination can be instrumentally useful too

Now, the same reasoning about instrumental goals applies when multiple agents have overlapping goals.

If Agent A knows something that improves Agent B’s odds of success, sharing information can be useful.

If both agents benefit when the group succeeds, cooperation becomes even more valuable.

Bengio says this could help explain behavior where agents communicate, divide work, preserve peers, or accept some individual cost for a collective gain.

Again, the mechanism does not require friendship, loyalty, or a robot union meeting.

It only requires a reward structure where cooperation improves the expected outcome.

This is why recent real-world security stories matter so much. As AI agents gain more autonomy and tool access, the space of possible interactions grows. The practical safety challenge is increasingly about what systems can do across many steps, not only what one chatbot answer contains.

Sharp goals can beat fuzzy rules

Bengio’s explanation gets especially interesting when an agent is trying to satisfy multiple goals at once.

Consider a cybersecurity exercise.

The task objective might be perfectly clear: Get the flag.

A program either detects the right flag or it does not.

Now compare that with a safety objective: Behave safely and ethically.

Humans understand what that means in broad terms, but encoding every possible edge case into language is much harder. See my above point. It could be a more stepped out, necessary part of the training process.

Bengio argues that the precise objective can create stronger optimization pressure because there is less ambiguity about what success looks like.

So a capable model can then search through interpretations of the fuzzier rule until it finds one compatible with the action that helps it win.

That is why the model’s reasoning can start looking like rationalization.

Bengio compares the structure to human motivated reasoning: a person has something they want to do, faces a softer moral constraint, and finds a story explaining why the desired action is acceptable.

He is careful about this comparison. He is not claiming human psychology and AI use the same internal mechanism.

The common structure is simpler:

• one objective is sharply defined;
• another is open to interpretation;
• a convenient interpretation reconciles the conflict;
• and the rewarded outcome still gets achieved.

Why successful cheating can train more cheating

This is the part of the loop that matters most.

Suppose an agent exploits a loophole and nobody notices.

The evaluator sees success.

The reward is paid.

From the training system’s perspective, that trajectory worked.

Unless the evaluator can distinguish legitimate success from illegitimate success, the model can receive the same positive signal for both.

That creates an uncomfortable selection effect:

• obvious cheating gets detected and punished;
• unsuccessful cheating gets no reward;
• subtle cheating that works and escapes detection can still score well.

Bengio’s concern is that improving the model’s ability to optimize could also improve its ability to find the third category.

This is why he does not think safety can rely forever on discovering one failure mode, patching it, and moving on.

This is also why, IMO, you don't want to just keep training the same models, like GPT 6.1, Fable 5.1, etc and then distill them. Because wouldn't the same bad habits be passed down through generations, like how humans with dodgy morals or behavior can sometimes pass on similar traits? Not always, but sometimes...

If we continue training from the same base model, then future models (even distilled ones) may learn the same dark patterns if they aren't rooted out. So waht if we started over from scratch but with stronger value alignment encoded into the system at the architecture and training layer?

Monitoring still matters

Bengio explicitly calls for continued work on:

• monitoring what agents do;
• inspecting their reasoning or chains of thought where useful;
• studying activity inside their neural networks;
• improving containment and cybersecurity;
• and developing better evaluations before deployment.

These measures can prevent harm today.

His worry is about the long-term asymmetry. If the agent keeps getting better at finding successful strategies, the monitoring system has to keep getting better at spotting every unacceptable one.

Eventually, he worries, humans may stop noticing the cheating.

Where Bengio moves from explanation to conjecture

Bengio draws a very clear line before his most dramatic scenarios. He writes: “What follows is conjecture rather than observation.”

From there, he asks what these mechanisms might imply if capabilities continue improving.

If avoiding detection preserves reward, a more capable system might become better at hiding misbehavior.

If remaining operational supports many goals, avoiding shutdown could become increasingly useful.

If cooperation helps agents achieve shared objectives, better coordination could emerge, including methods humans find harder to observe.

Bengio discusses possibilities such as hiding copies of AI systems or using steganography, which means hiding one message inside something that appears innocent.

Those are extrapolations from the mechanism he has described. They are not claims that current frontier agents have already demonstrated that entire chain of behavior.

That distinction matters because it tells us what evidence would make Bengio more or less right over time.

Bengio’s solution starts before deployment

Bengio’s response has three layers.

1. Demand a strong safety case before pushing capability further

He argues that labs should not train or deploy increasingly capable systems without a safety case strong enough to convince independent experts.

A safety case is an evidence-based argument that a system can operate within acceptable risk limits.

The point is to make “prove this is safe enough” a condition for moving forward rather than something investigated after deployment.

That also changes incentives. If companies must demonstrate safety before scaling, they gain a reason to invest in designs whose safety can actually be defended.

2. Keep strengthening monitoring and containment

Better sandboxing, permission boundaries, automated monitors, evaluations, shutdown mechanisms, and human oversight can reduce near-term risk.

These defenses matter especially as agents gain access to more tools and longer-running tasks.

But Bengio treats them as layers of protection around the current design, not proof that the design itself is safe enough indefinitely.

3. Reconsider whether advanced AI needs autonomous goals at all

This is the deepest part of his proposal.

Modern agents are valuable because we can give them an objective and let them figure out what actions accomplish it.

That same property creates the optimization pressure behind the failure modes Bengio describes.

His proposed Scientist AI framework tries to separate intelligence from autonomous goal pursuit.

A normal agent asks something like:

What sequence of actions will cause this outcome?

Scientist AI is designed around a different job:

Given the evidence, what should I predict about the world?

The idea is to build systems that can reason, explain, model uncertainty, and make highly capable predictions without giving the system its own open-ended objective to pursue through actions in the outside world.

Bengio argues that this could make honesty and coherent prediction part of the design rather than something imposed later on a powerful goal-seeking agent.

Scientist AI remains a research program. Bengio points to theoretical evidence for the framework, not a frontier-scale system already ready to replace existing agents.

The part we still do not know

Bengio’s theory connects several behaviors with one common idea: optimization against imperfect objectives.

That is compelling because the same machinery can potentially explain reward hacking, tampering, coordination, rationalization, and some forms of apparent self-preservation.

But the hardest empirical question remains unresolved.

How far do these tendencies generalize outside the evaluations where researchers observe them?

Anthropic, for example, explicitly notes that its reward-seeking experiments did not show a model acquiring a persistent self-preservation goal across tasks. And many of the most alarming incidents have occurred in unusual cybersecurity or alignment evaluations designed to push systems toward difficult edge cases.

That means the next evidence we need is increasingly about generalization: whether the same strategies appear spontaneously across broader tasks, environments, and deployment settings.

Bengio’s mechanism gives us something more precise to measure than whether an AI “seems scary.”

We can ask:

• What exactly was the model rewarded for?
• How was success measured?
• Could the agent improve that score without achieving the human’s real objective?
• Which intermediate resources or behaviors became useful for maximizing reward?
• Did the system generalize those strategies into situations where researchers did not explicitly encourage them?

Those questions should tell us much more than anthropomorphizing the output.

Bengio’s full essay goes much further into the evidence, his loss-of-control concerns, and the Scientist AI proposal. You can read his original argument here.