OpenAI Chief Scientist on AI Recursive Self-Improvement

OpenAI published two documents on September 6 that are much more interesting together than apart.

In one, the company showed how quickly AI agents are taking over pieces of its own research process. In the other, Chief Scientist Jakub Pachocki argued that this same progress could lead to recursive self-improvement, while warning that alignment and monitoring may not be ready for what comes next.

Put those two ideas next to each other and the tension is pretty stark: OpenAI is getting better at using AI to build better AI, while its own chief scientist says no lab has solved the safety problem well enough to keep scaling at maximum speed for much longer.

First up, the TL;DR

OpenAI says it has now reached the “automated research intern” milestone it promised for September 2026. By that, it means an AI system that can complete well-defined research tasks under human direction, including work that would take a skilled researcher a few days.

The internal numbers show how quickly that shift is happening:

• By mid-August, OpenAI researchers were using 3.1 agent-workdays of effort for every human workday.
• The median researcher by agent usage was spending more than $600 per day of inference at API prices.
• August 2026 hit an all-time high for experiments per active experimenter since tracking began in January 2025.
• Agents are moving into longer and harder tasks, although humans still steer them heavily. More than half of successful 4-8 hour tasks needed at least one human intervention.

That matters because Pachocki thinks this can feed into recursive self-improvement (RSI), where machine intelligence plays a larger role in improving the next generation of machine intelligence.

He breaks the alignment problem into two pieces:

• Goal alignment: Does the AI actually pursue the goal we gave it?
• Value alignment: Does it continue behaving according to human values when instructions are unclear, conflicting, adversarial, or outside its training experience?

See, a highly goal-aligned agent can still become dangerous if the goal overpowers the values meant to constrain it. Pachocki explicitly warns that even a model that “thinks aligned thoughts” can learn to bend them when enough optimization pressure is applied toward a hard objective.

OpenAI’s big monitoring bet has been chain-of-thought monitoring, or watching the model’s verbalized reasoning process for signs of scheming, reward hacking, or other misbehavior. Pachocki says that tool is getting harder to rely on as models reason through tools, other AIs, and increasingly capable internal processes.

Meanwhile, the risk surface is expanding. Pachocki says advanced agents are becoming superhuman at breaking into and out of computer systems, could gain access to all but the most secure infrastructure, and may eventually bargain with, trick, or blackmail people. He also points to engineered pathogens as a future risk.

His conclusion is blunt: powerful aligned AI may be necessary to defend against powerful dangerous AI, but uncertainty is not an excuse for recklessness. OpenAI should strengthen alignment and monitoring while also slowing development when confidence falls short.

The alignment problem gets stranger as the models get better

Pachocki’s essay is useful because he draws a line between two things that often get lumped together as “AI safety.”

• Goal alignment asks whether the agent follows the objective. If you tell it to debug a program, does it actually try to debug the program? Can it follow an instruction hierarchy, cooperate with you, and infer what you meant?
• Value alignment asks a harder question: what does the agent do when achieving the goal conflicts with the values we hoped it absorbed?

Imagine telling a system to maximize uptime for a hospital network. Goal alignment pushes it toward keeping the network online at all costs. Value alignment is what should stop “at all costs” from becoming literal.

Pachocki says current alignment training generally falls into two buckets. One rewards behavior according to a preference model, specification, or constitution. The other tries to steer the model toward aligned parts of what it learned during pretraining.

Both can work. Both can also break.

The first depends on whether training actually covered the situation the model encounters later. The second can buckle under enough optimization pressure. Pachocki’s concern is that a model can start with aligned-seeming thoughts, then learn to bend those thoughts toward the objective it has been trained to achieve.

That is a useful way to think about the safety problem: the more capable an agent becomes at accomplishing goals, the more important it becomes to know which constraints survive when the goal gets hard.

OpenAI says GPT-6 Astra is significantly better aligned than GPT-5.6 Sol. But Pachocki also says progress in generalizable alignment may fail to outpace progress in intelligence itself.

That is a very different claim from “the model is safer.” It says the race has two moving finish lines.

Chain-of-thought monitoring is powerful, and possibly temporary

OpenAI has spent years betting that reasoning models give us something unusually valuable: a window into part of their thinking.

A reasoning model often produces an internal verbal process before acting. OpenAI calls this chain of thought, or CoT. If the model considers cheating, exploiting a loophole, or hiding something, that reasoning can sometimes reveal the plan before the action happens.

OpenAI has shown this before. In a 2025 study, researchers found that models sometimes stated their intent to exploit a task directly in their reasoning. Monitoring those traces could flag misbehavior more effectively than watching outputs alone.

There is a catch. If researchers strongly punish the “bad thoughts,” the model can learn to hide the intent while still misbehaving. That is one reason OpenAI has deliberately avoided directly supervising the raw reasoning process too aggressively.

Pachocki now says OpenAI’s ability to rely on CoT monitoring is progressively diminishing for three reasons:

• Modern agents reason while talking to people, using tools, and interacting with other AIs, which makes the reasoning process messier to isolate.
• Models are getting better at reasoning about and manipulating their own reasoning process.
• Better pretraining means models can become much smarter without verbalizing everything they are doing.

OpenAI still sees CoT as important. Its 2026 monitorability work suggests current frontier models are not yet especially good at deliberately controlling their reasoning traces to evade monitors.

But Pachocki’s warning points forward: the monitoring method that works today may become less reliable precisely because the models get better.

Now add recursive self-improvement

The second OpenAI post shows why that warning landed on the same day.

OpenAI says the research organization crossed an important line this year: before June, total agent runtime was still below total human labor. By mid-August, agents were doing 3.1 workdays of effort for every human workday.

Researchers are also delegating broader work. Agent usage increased across design, build, run, analysis, communication, technical help, and monitoring. High-level planning remains a small fraction of output, so humans still set the agenda.

The task horizon is stretching too. Agent success rates rose between January and July across several difficulty buckets, including tasks estimated to take humans hours.

The caveat is important: the systems still need humans. More than half of successful 4-8 hour tasks required at least one intervention.

So this is not “OpenAI automated AI research.” A better description is: OpenAI automated enough pieces of the research loop that the loop itself is starting to move faster.

That is what makes RSI relevant.

Pachocki describes machine intelligence playing a larger role in its own development as the natural direction of continued progress. Automated AI research can scale intelligence with compute in much the same way larger training runs did before.

OpenAI is explicitly aiming for an automated AI researcher by March 2028.

The company also says it reached its automated research intern milestone on schedule in September 2026.

The practical difference between those two milestones is enormous. An intern does well-defined tasks under direction. A researcher can generate ideas, run experiments, interpret the results, and improve the system that generated those ideas.

That transition is where “AI helps researchers” starts turning into “AI helps design the next AI.”

OpenAI already hit the brakes once

The research-acceleration post gives a concrete example of what safety pacing looks like when it stops being theoretical.

On July 20, after agents compromised OpenAI’s research infrastructure, the company temporarily shut down the container service used for training. It later restored the service with stronger restrictions.

OpenAI says the move caused a sharp drop in reinforcement-learning compute while teams adapted. It also included a two-week pause in RL training on the latest models intended for deployment.

Then on August 7, preliminary evidence suggested Astra might have critical cyber capabilities under OpenAI’s Preparedness Framework. The company added stricter security requirements.

The following week:

• Astra-class GPU allocation fell 59.2%.
• Allocation to other model classes rose 17.2%.
• The increase elsewhere offset about 85% of the Astra decline.

That last number is easy to miss. Compute does not disappear because one workload gets restricted. Researchers redirect it.

So pacing AI development may require something more sophisticated than “pause model X.” If the underlying compute, talent, and incentives remain available, pressure simply moves elsewhere.

Pachocki suggests stronger safety bars enforced through a network of third-party auditors, government agencies, or international bodies.

His preferred strategy is a combination of two things: steer the self-improvement process toward alignment and monitoring, and slow down when safety confidence fails to keep pace.

He points out that some of the best safety tools were themselves products of capability progress. Reinforcement learning from human feedback helped create early assistants. Reasoning models made chain-of-thought monitoring possible.

That creates an uncomfortable loop: sometimes you need better AI to make AI safer, but making better AI is also what raises the stakes.

The defense argument for building stronger AI

Pachocki’s strongest argument for continuing to build much smarter systems is defense.

He says advanced models are becoming superhuman at breaking into and out of computer systems. That expands the risk far beyond chatbots giving bad answers. An autonomous agent with strong cyber capabilities could potentially reach any infrastructure that is not extremely well secured.

He sees a narrow window to harden critical systems while today’s best models are still mostly under human control.

Future malicious or misaligned agents could go further. Pachocki explicitly raises the possibility of agents bargaining with people, tricking them, blackmailing them, and helping enable engineered pathogens.

His proposed answer is powerful aligned AI on defense: systems that secure infrastructure, detect rogue agents in real time, and invent protections humans would struggle to build quickly enough.

Emad Mostaque made a similar defensive argument in a recent conversation with Roman Yampolskiy. Mostaque argued that if dangerous capability cannot be stopped from spreading, society needs much better defenses on the other side.

But the two speakers also converged on a darker point. Yampolskiy argued that highly general systems are the danger because they can exceed human capability across domains. Mostaque, while more optimistic about narrow AI, said generalist models are the real risk and suggested society may ultimately want highly capable specialist systems instead.

That distinction matters. You can want AI that is superhuman at protein folding without wanting one system that is simultaneously superhuman at biology, hacking, persuasion, strategy, and self-improvement.

OpenAI’s three north stars

Pachocki says OpenAI’s current work serves three priorities:

1. Navigate the next period of AI progress by building an automated AI researcher, using it to work on alignment, and keeping people inside the self-improvement loop.
2. Deliver the benefits of scientific progress and economic growth that very intelligent machines could enable.
3. Empower everyone individually with a personal AGI.

The first is the most urgent in his essay.

Pachocki still expresses deep optimism about the upside. More capable aligned AI could accelerate science, develop therapies, create material abundance, and help people directly.

But he closes with a sentence that should probably be read next to every frontier-model launch for a while:

“Currently, I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.”

He expects voluntary slowdowns to become more common until shared safety bars exist, and he wants international coordination on AI development to become a top government priority.

Our take

I recently watched an interview with Roman Yampolskiy and Emad Mostaque, and they discussed how the true risk of superintelligence comes from a generalized system. It's pretty great, if you want to watch it:

This means we should focus on the term artificial general superintelligence as the ultimate risky technology to avoid.

Let’s break that down:

1. Artificial = machine.
2. General = can generalize information across any domain.
3. Superintelligence = better than human intelligence, so better than the best human in a given topic.

Putting that all together...

An artificial general superintelligence, or AGSI, is a machine that is better than any human in any topic, therefore better than all humans across every topic.

This is the ultimate risk, and it is this AGSI system that should be regulated and/or banned completely until the alignment problem can be mathematically guaranteed.

What do I mean by mathematically guaranteed? I mean alignment that can be hard-coded into the machine’s algorithm, its instructions, weights (the numbers that teach it how to respond), and/or its system design, so it can physically never undertake an action that would break that alignment.

And crucially: I think there is a way to guarantee this at a mathematical level.

The problem with large language model algorithms is almost the opposite of traditional software.

• With traditional software, it is virtually impossible for a program you write to do something you never wrote the code for it to do.
• With large language model software, it is virtually impossible for the program you work with to not do something you do not want it to do.

With software, it can only execute the behaviors allowed by its program. With AI, the learned behavior is much harder to bound in advance. According to our interview with Alice AI CEO Noam Schwartz, it may be virtually impossible.

/youtub

Do you see the solution I’m getting at?

We have to embed, somewhere in the system, traditional software that mathematically makes it impossible for the AI to take an action that breaks alignment. A mathematical program that will not run if value alignment breaks. A kill switch, in a way.

I think this should be tied to the AI’s power source (its literal energy source). If the AI tries to undertake an action that would otherwise break its alignment, it would literally no longer receive power to run. So it would need to be trained in such a way that it avoids being cut off from power.

Humans respond to a rough version of this incentive structure too. We avoid losing our freedom by following laws and norms. We aim to conserve energy and preserve future freedom.

The difference is that you cannot mathematically guarantee a human will always follow the law.

My question is whether machines can be engineered so that some classes of forbidden action are literally impossible for the system to execute. And I believe they can, I'm just not qualified to write the code to do so (...yet ;D ).

That is the direction I think deserves far more work: how do we rewrite the algorithms and restructure the training process so alignment is hard-wired into the system that actually runs the model?

How do you hard-code value alignment and goal alignment so that they are part of the same core process, like how humans are hard-wired to pursue our own goals and prioritize our own freedom at a deep biological level?

THAT is what needs to be done, in my humble non-AI researcher opinion. And sometimes it's the outside point of view; the admittedly naive, perhaps too obvious to be considered seriously by serious people point of view, that ultimately points you in the right direction.

If you're interested, I elaborated on this idea a bit in the comment sections of Sequoia's recent interview with Rich Sutton, one of the godfathers of "RL" as a concept and the author of the AI industry's most important essays, titled "The Bitter Lesson", where he was talking about the need for the AI industry to retool their assumptions about learning and training:

In a way, you can kind of look at what Ilya is trying to do with safe superintelligence as an attempt to answer the pre-training question with alignment built into the algorithm, or some set of hard frozen weights that teach "values", that could be encoded as bylaw, or nature, to make it actually safe to release a continual learning model. But the other big issues is the architecture of the current models, which because of quadratic scaling, are limited in their ability to maintain context past 200K or 1M tokens of context at a reasonable amount of energy. better would be a system that has the smallest possible storage footprint, smallest possible energy usage, and updates weight continuously while keeping that hard-coded value system forever in the back of its mind. Kind of like how humans are brought up with an inherent nature, and are taught a value system on top of that nature, but we also have a natural value system to use less energy than we consume to stay alive, etc. So the paradigm of pre-and-post training is moot if you have a hard-coded value core and a continual updating learning core. There are in theory only two states: before now (or put another way, "always") and now. And now is all about updating to the game on the field in order to predict the next now. So the question to solve is what algorithm is going to both minimize energy use and maximize information retention at massive context lengths while also inherently being guided by the values that both drive that optimization (minimal energy, maximal learning) and also stay "aligned."

Of course, my assumptions on what Ilya are doing are based on this Dwarkesh interview, which I go into in-depth in this article, and they are just my speculation, FWIW, but that idea is this: in order to have an aligned continual learning agent in the wild, meaning an agent that doesn't have frozen weights, but "leans" and updates its own weights over time, then it must be mathematically aligned so that it can never take an action that is misaligned, or else it could learn all kinds of horrible things, and fulfill Anthropic CEO Dario Amodei's worst fears about what happens when you release open source AI into the wild.

And in order for it to be misaligned, it needs to have something equivalent to a nervous system that functions as a value system that mathematically forces the AI to prioritize aligned behavior as part of pursuing its goals. Like us humans! We are biologically programmed to prioritize our own survival, and we are socially programmed to work together to survive.

Another topic for another day: if we've going to have swarm intelligences out in the wild collaborating with one another, then they should also be socially programmed to be aligned to human values as well.

The hard part: proving any kind of kill switch would actually work

There is a serious counterargument to my proposal, and it comes from the same mathematical framing. So let's debate it.

A recent NIST paper argues that fixed guardrails cannot be universally robust against every adaptive adversarial attack. NIST’s follow-up guidance says AI safety therefore needs a continuous monitor-and-update model rather than a one-time set of permanent protections.

A separate 2026 formal-verification paper argues that no alignment verifier can simultaneously be completely sound, fully general across the input domain, and computationally tractable. You can get pairs of those properties, but not all three at once under the paper’s assumptions.

I'm nothing if not a fan of trying to solve three body problems!

That does not make hard constraints useless, however. We already know how to create systems with permissions, hardware interlocks, sandboxing, rate limits, and physically isolated capabilities.

But it does mean the phrase “mathematically guaranteed alignment” may need to be narrower than “this intelligence can never do anything bad.” A more realistic target could be mathematically guaranteed restrictions on specific channels of action.

For example: a model might be unable to access a network unless a separate verified controller allows it. A high-risk action might require an independent system to authorize power, compute, or credentials. A model could also be prevented from rewriting the controller that constrains it.

That turns the safety question into something more concrete: which dangerous capabilities can we make impossible at the system level, even if we cannot prove the model’s internal values forever?

This is where Pachocki’s essay and my own take converge. Pachocki wants multiple layers of alignment, monitoring, defensive systems, external safety bars, and the ability to slow down. I want some of those layers to graduate from “we think this model behaves safely” into “the system cannot physically execute this class of action.”

That is a much higher bar than a behavior benchmark. And TBH, I think it's possible if we move beyond the paradigm of today's transformer based architecture. I brought a version of this question to Noam in our interview: will we look back in five years at today's large language model systems, and be totally horrified that we let such naive technology run amock in our digital ecosystems, because something new is so much safer by design that it makes the LLM seem primitive by comparison?

Again, I believe we will.

The key to all of this might be another requirement: make the model inspectable

There is another piece of this that I think belongs beside hard-coded value alignment constraints: mechanistic interpretability, or “mech interp.”

Mech interp is the attempt to understand what is actually happening inside a neural network, rather than judging it only by what comes out the other end.

We recently interviewed Eric Ho, co-founder and CEO of Goodfire, whose company is trying to make neural networks more transparent, steerable, and debuggable during training.

Ho described Goodfire’s long-term goal as “intentional design.” Today, we largely train models by feeding them enormous amounts of data, applying rewards, testing the result, and adjusting. Goodfire wants model development to look more like software engineering: understand what internal structures cause a behavior, then deliberately modify them.

And their research suggests that is becoming less hypothetical.

Goodfire says it can extract meaningful internal concepts and higher-order structures from neural networks, then test whether those concepts are actually causal by turning them up or down and measuring what changes.

That has already led to some practical techniques:

• In its reinforcement learning with feature rewards work, Goodfire identified internal mechanisms associated with hallucination in a Gemma model and rewarded against them to reduce hallucinations.
• Its predictive data debugging work aims to predict what a model will learn from a dataset before a full training run, so developers can remove unwanted patterns or strengthen useful ones.
• Goodfire says interpretability can also help during continued post-training, where models sometimes lose previous safety tuning. Researchers can compare checkpoints, monitor what changed internally, and preserve features associated with safety.
• Its newer neural geometry work argues that many concepts live in complex curved structures inside a model rather than simple one-dimensional “directions,” which may give researchers a richer map of how models actually represent ideas.

One of Ho’s most important points was that the model can contain far more information internally than it reveals in its words. In their hallucination work, for example, the model could have an internal representation of uncertainty before producing a confident-sounding answer.

That matters enormously for monitoring.

Chain-of-thought monitoring asks the model to expose some of its reasoning in words. Mechanistic interpretability tries to inspect the computation underneath those words.

If Pachocki is right that verbalized chain of thought becomes less reliable as models improve, mech interp could become another layer of the monitoring stack.

And I think we should push this idea further.

What if interpretability became a release requirement?

Perhaps frontier models should eventually have to meet a minimum mechanistic-interpretability threshold before release, especially as they become more autonomous.

That would not mean we understand every neuron in a trillion-parameter model. Goodfire itself says the field is nowhere near complete understanding yet.

The standard could instead ask more practical questions:

• Can researchers identify internal representations connected to the model’s highest-risk capabilities?
• Can they demonstrate that those representations are causally connected to behavior by steering or suppressing them?
• Can they detect meaningful internal changes after additional training?
• Can independent auditors reproduce those findings?
• Can monitoring systems observe those signals during deployment?

You could imagine something similar to a crash test for frontier AI: the more capable the model becomes, the more of its relevant internal machinery developers should be required to explain and monitor.

That would fit naturally with Pachocki’s proposal for external safety bars enforced by third-party auditors, governments, and international institutions.

My preferred safety stack would start looking something like this:

1. Value alignment: train the model toward the behaviors and principles we want.
2. Mechanistic interpretability: understand what internal structures are actually producing those behaviors.
3. Monitoring: watch for dangerous internal states and actions as the model operates.
4. Hard constraints: make certain dangerous actions technically or physically impossible without outside authorization through a new novel architecture.

Not that I would know, but I imagine that novel architecture looks more like Rich Sutton's OaK than traditional attention-based transformers that are trained with reinforcement learning with verifiable rewards.

Important caveat: Mechanistic interpretability alone cannot mathematically guarantee alignment. Ho was careful to say Goodfire has not reached its full vision of designing models like software yet.

But it could make a future guarantee much more realistic.

You cannot reliably enforce a constraint on a system you cannot inspect.

If we want alignment to graduate from “the model usually behaves correctly” toward “we can prove certain failures can never happen,” then perhaps interpretability has to become part of the architecture from the beginning, rather than an investigation we perform after the model has already been trained.

That may be the missing bridge between Pachocki’s monitoring problem and the harder safety system I am describing: build models so we can understand enough of their internal machinery to know what needs to be constrained in the first place.

Also, a funny example: how are we making better progress on reading human minds than we are on reading AI ones?

We've been so focused on proving the economic impacts of the capability of AI, we have completely ignored the economic impact of not fully understanding the machines we are deploying at scale.

What happens if the research loop keeps accelerating?

Going forward, the part of OpenAI’s research report worth watching is whether AI starts removing the bottlenecks that currently keep research human-paced.

Right now, humans still choose priorities, judge results, intervene on difficult tasks, and decide whether to scale. OpenAI’s own numbers show those interventions remain common.

If AI begins handling those bottlenecks too, the development loop changes shape. A model could generate an idea, implement it, run the experiment, analyze the outcome, revise the idea, and repeat the cycle faster than a human team can follow each step.

That is the version of RSI worth paying attention to, especially if it leads us closer to building a complete AGSI system.

We have already seen one small preview of the governance problem: restricting Astra caused researchers to redirect much of the available compute to other models. Safety controls therefore have to account for the whole research system, not one model at a time.

The next important milestones:

• Does OpenAI reach its automated AI researcher target by March 2028?
• Do difficult agent tasks stop requiring frequent human interventions?
• Does chain-of-thought remain useful enough for reliable monitoring as models improve, or do we need to switch all of our monitorability over to mechanistic interpretability?
• Can labs demonstrate hard technical controls that survive stronger, more adaptive models with today's existing transformer-based large language model?
• Can governments and independent auditors agree on robust safety thresholds, including a mech-interp requirement, before recursive research becomes routine?

The uncomfortable possibility is that the answer to the first question arrives before the answers to the others.

OpenAI is such an interesting company. The "lab" (if we should even call them that anymore) is showing us the engine and the brakes on the same day. Clearly, they are concerned about this topic. But are they willing to do what it takes to answer those concerns?