AI Recursive Self-Improvement, Explained: The Real Bottlenecks

If you want to understand the whole “AI builds better AI until everything goes vertical” argument without immediately wandering into science fiction, this conversation is one of the better places to start.

In his latest interview, Dwarkesh Patel sat down with Beren Millidge, CTO of Zyphra; John Schulman, chief scientist at Thinking Machines and former OpenAI co-founder who helped lead the RLHF work behind ChatGPT; and Charlie O'Neill, head of model training at Baseten.

Instead of beginning with “when do we get superintelligence?”, Patel asks a much more useful question: (0:49)

Imagine we get to 2036 and the world still looks surprisingly normal. What went wrong?

That question forces the whole recursive self-improvement debate down from the clouds and into actual engineering.

Because an intelligence explosion (of which there are three kinds) could start much earlier than the point where a model feels godlike. Automate enough of the AI research loop, and each generation can help build the next faster than humans can keep pace. (3:05)

The conversation is really about what stands between today's models and that loop. And surprisingly, “make the model smarter” is only part of the answer.

This also lands at an interesting moment. OpenAI says it has already reached its “automated research intern” milestone and is aiming for an AI researcher in 2028, which we broke down earlier this week.

First, what is recursive self-improvement?

Recursive self-improvement, or RSI, is the idea that AI begins helping improve the process used to build better AI. Then the improved AI helps even more. Then that one helps build something better again.

Keep tightening that loop and AI progress could accelerate because the researchers improving the technology increasingly include the technology itself. (3:05)

People sometimes imagine this as one model literally opening its own brain, editing a few files, pressing save, and becoming superintelligent before lunch. Maybe you could train the Fruit-fly brain to do that, but probably not a large language model.

The version discussed here is much more believable because pieces of the loop are already familiar.

An AI researcher could:

• read papers and previous experiments;
• write and debug training code;
• design smaller experiments;
• analyze results;
• propose the next experiment;
• build training environments;
• generate or filter training data;
• evaluate models;
• turn successful discoveries into training data for the next model.

Humans already use AI for some of those jobs.

RSI happens when enough of them connect together that the system starts driving the research process itself.

And that last jump, from doing research tasks to choosing the research direction, is where this conversation gets really interesting. (13:40)

Models are getting very good at doing things we can grade

One of the strongest cases against rapid RSI starts with something we've all experienced.

A new AI model comes out. Everyone uses it for a week and loses their mind. This is it. We have arrived.

Then you spend a month with it.

Suddenly you know exactly where it's dumb.

Maybe it can write 10x more code than you, but it misses a requirement. Maybe it finishes a complicated analysis and then fails to notice its conclusion contradicts the spreadsheet. Maybe it can execute six hours of work but makes one terrible judgment halfway through that ruins the whole thing.

Schulman describes this as a recurring cycle: models catch up to humans in one area, then the workflow gets bottlenecked by whatever weakness remains. (2:12)

That could keep happening.

The deeper technical version is a generalization problem. Generalization is the ability to learn something in one setting and successfully apply it somewhere different. (9:39)

Today's labs can make models extremely good at tasks where success is easy to measure. Coding is perfect for this. Give the model a program, run the tests, and you can often determine whether it worked.

Math is another great example. There is usually an answer.

That lets labs use reinforcement learning, or RL, where the model tries different approaches and gets rewarded when it succeeds. Do that across huge numbers of problems and you can strengthen useful behaviors. (34:08)

Real life has an irritating habit of refusing to provide an answer key.

The bottleneck may be choosing the goal

This is probably the most important distinction in the whole conversation. (7:10)

Imagine I tell an AI researcher:

Get this model's training loss below 1.3 using less compute.

Now we have a clean objective. The AI can try different architectures, learning rates, data mixtures, optimizers, and training schedules. It can run experiments and compare the results.

The panel thinks AI could become extremely good at this type of research.

They give examples where more analysis might already have saved researchers months or years. One involves old scaling-law work, where an AI carefully inspecting intermediate checkpoints might have caught an issue around training schedules much earlier. (12:40)

Take that idea much further.

Before launching a multimillion-dollar experiment, you could have thousands of AI researchers spend the equivalent of decades testing smaller versions, reading every relevant paper, running ablations (experiments where you remove one component to see what mattered), and arguing over the optimal setup. (10:48)

Then you run the expensive experiment.

Afterward, another army of agents analyzes every result before deciding what comes next.

The amount of thinking between experiments could explode.

But now change the prompt:

Figure out what the next important breakthrough in AI should be.

Uh oh.

The objective disappeared.

Research often begins with an intuition that something important is missing. Researchers decide which question is worth asking, invent an experiment that might reveal an answer, interpret ambiguous results, and sometimes realize they were asking the wrong question altogether. (31:52)

Schulman's answer for the human job that may last longest is basically this: deciding what we actually want. (15:55)

Or, as the panel eventually puts it, “Alignment is the final job.”

That applies beyond AI safety. Somebody still has to specify what “better” means.

That's the actual RSI threshold

A fully automated research loop needs to do more than optimize objectives.

It needs to repeatedly:

1. Choose a useful objective.
2. Figure out how to measure progress toward it.
3. Improve against that measurement.
4. Recognize when the measurement itself is wrong.
5. Choose the next objective and repeat.

That loop is exactly where the panel locates the hard part of RSI: propose objectives, optimize them, evaluate what happened, and decide what objective comes next. (13:40)

The first three are increasingly familiar.

Four and five are harder.

This is why “AI can code better than humans” and “AI can recursively improve itself” are separated by more distance than they initially appear.

Coding an experiment is execution. Deciding that the experiment is worth running is judgment. Patel asks what humans may do last before AI automates AI R&D, and the answer is basically deciding which experiment is worth running next. (14:58)

Today's models can produce endless research ideas, but the panel notes that those ideas often look like lots of tiny miscellaneous steps. Human researchers still provide much of the taste involved in deciding which direction could actually unlock something important.

Taste sounds squishy until you see it fail.

A senior software engineer knows a clever architecture may become a maintenance nightmare six months later. An experienced researcher knows one weird result deserves three more experiments while another is probably noise. (49:58)

Those judgments often come from watching choices play out over long periods.

And that gives us another potential bottleneck: time.

AI research may actually be easier than being your coworker

This sounds backwards, but it is one of my favorite observations from the discussion.

Some jobs are cumulative.

AI research might be one of them. (47:18)

Once researchers discovered the Transformer, the next generation did not have to rediscover the Transformer. Once mixture-of-experts worked, you could add it to the training stack. Once somebody figures out a better reinforcement-learning method, future training runs inherit that discovery. (47:32)

Progress piles up. It compounds. Like interest! Which also relates back to the problem with keeping research private at these private labs... it begets centralization. But that's another topic!

Compare that with working inside a law firm.

Your model needs to know who handles which client, what happened in yesterday's meeting, which partner hates a certain document format, where somebody moved a file, and why Susan from accounting has suddenly changed the process everyone used last month.

The environment keeps moving. (48:20)

That is called a non-stationary distribution: the world you're learning about keeps changing underneath you.

This creates the wonderfully strange possibility that automating AI research could be easier than completely automating many normal office jobs. (48:43)

Your future paralegal needs to keep learning an evolving human organization.

Your future AI researcher can spend much more of its time improving a cumulative technical artifact.

As one speaker jokes, it would be rather unfortunate if recursive self-improvement turned out to be easier than being a paralegal.

Unfortunately, there is a real technical argument for it.

Then there is the sim-to-real problem

Labs currently solve much of the training problem by building simulated environments.

Want a model to become a better coding agent? Give it thousands of simulated repositories, bugs, tools, and tasks.

Want it to learn finance? Build financial environments.

Spreadsheets? Throw those in too.

This is sim-to-real: train useful behavior in environments you can generate and control inside a data center, then hope those skills transfer when the model encounters messy reality. (38:48)

The panel sees the frontier moving from coding into finance, presentations, and eventually the long tail of knowledge work. (35:30)

That approach works because simulated experience is cheap to generate at scale. You cannot ask a human manager to participate in 100,000 reinforcement-learning episodes so a model can practice being a coworker.

The data center can happily simulate 100,000 fake managers.

The problem grows with task length. (45:27)

A six-minute coding task can stay inside a sandbox. A six-month business project eventually requires customers, coworkers, negotiations, changing goals, surprises, and consequences that are difficult to reproduce.

Running a profitable business is hard to simulate.

Winning a court case is hard to simulate.

Being a useful employee for a year is hard to simulate. (45:51)

The longer the horizon gets, the more training starts bumping into reality.

We've written about this before from another angle. Our breakdown of reinforcement-learning environments looked at why frontier models can ace benchmark-style problems and still struggle with messy workplace tasks.

Nick Heiner of Surge AI explains "RL environments" more in this interview, too.

Eventually the model needs to learn from doing the job

This is where continual learning (Dwarkesh's favorite hobby horse) enters the story.

Think about it like this: Humans learn while deployed. Tech jargon I know, but bare with me, I'll explain it.

What's "while deployed" mean? You do something. You screw it up. Somebody explains what happened. You remember the lesson tomorrow.

Today's AI systems mostly operate differently.

You can have millions of copies of a model working across millions of tasks (this is also being "deployed"), YET... the underlying model does not instantly absorb everything those copies experience.

That seems crazy when you think about the potential scale.

Humans get one lifetime.

A widely deployed AI system could collectively accumulate the equivalent of millions of years of experience across its instances. (41:30)

Dwarkesh's panel thinks the learning loop will probably tighten in stages.

Today, deployment data can inform a later model generation. A company gathers useful traces, filters them, builds training environments, post-trains another model, and redeploys it. (42:15)

Eventually the refresh might happen every few months.

Then every few weeks.

Then every day.

Then every hour. (54:36)

At some point the distinction between “deployment” and “training” starts getting awfully fuzzy.

The “holy grail” would be a model that updates organically from each experience instead of waiting for the next generation. (43:00)

There's one annoying problem: learning new things can make models forget old things

Continual learning sounds obvious until you try it.

One common technique is supervised fine-tuning, or SFT. You show the model examples of behavior you want and update it to imitate them.

Do this repeatedly on a narrow stream of new data and models can suffer catastrophic forgetting, where learning the new material degrades capabilities they previously had. (55:39)

Imagine somebody learning French so aggressively that Excel disappears from their brain.

RL can be gentler because it often moves the existing model by comparatively small amounts. That helps preserve old capabilities, but also limits how much entirely new knowledge you can cram in. (56:31)

The panel's view is that this seems partly like a technique problem rather than proof that models lack capacity. (57:03)

I argue, many times on this very website in fact, that the issue here is the transformer architecture which scales quadratically and can't maintain more than 1M tokens in its context window at any given time at reasonable scales worth serving to customers. What's needed is a new architecture, and if we're fixing memory, we might as well fix alignment, which I talk about more here.

If you took all the accumulated information and trained a fresh model on it properly, the new model could be better. What we still struggle with is taking one existing model and updating it forever without slowly breaking something.

That tension is sometimes described as plasticity versus catastrophic forgetting: learn enough to stay adaptable without overwriting what already works. (57:28)

Our earlier deep dive on continual learning covers several attempts at solving exactly this problem.

And this technical detail matters enormously for AI timelines.

A model that learns safely from every deployment could improve on a radically different cadence from one that needs a giant retraining cycle every few months.

Better data may have mattered more than better architecture

Another fascinating part of the conversation concerns where recent AI progress actually came from.

We spend a lot of time talking about architectures: Transformers, mixture-of-experts, new attention mechanisms, bigger networks.

Beren describes an experiment comparing training recipes and datasets from different generations. (1:06:18)

At the small scale they tested, improved data produced roughly a 12x compute-efficiency gain, while architecture improvements produced about a 3.7x gain. (1:06:48)

That absolutely does not mean architecture is irrelevant.

Architecture can unlock an entirely new regime. Better attention mechanisms, for example, can make extremely long context windows practical. Once the architecture opens that door, however, the quality of the data you feed through it becomes hugely important. (1:08:44)

And AI is running into a weird data transition.

During pre-training, useful information already exists. The internet contains books, code, articles, conversations, documentation, and countless examples of humans doing things.

The job is largely finding signal inside all that noise. (1:05:30)

At the capability frontier, the answer may literally not exist in the dataset.

There is no secret completed proof of every unsolved math problem hiding somewhere in Common Crawl waiting for a better filter.

Somebody has to generate new signal. (1:05:52)

That can come from humans.

It can come from experiments.

It can come from deployment.

Or increasingly, it can come from models interacting with environments and producing new successful trajectories.

Which brings us back to RL.

Why reinforcement learning works better than it sounds like it should

There is a wonderfully counterintuitive bit here.

RL can receive surprisingly little information from each attempt. (1:18:14)

Suppose a model tries a difficult problem and receives a reward that basically says:

Worked.

or

Didn't work.

That is a tiny amount of information compared with the thousands of tokens inside the model's reasoning.

So why does RL produce such dramatic improvements?

One answer from the panel is that the model arrives at RL much more prepared than people assume.

Before RL, labs can perform mid-training, using carefully constructed data to teach the model relevant skills and reasoning patterns. Beren estimates that this can sometimes get a model roughly 80% of the way toward the final RL-trained system. (1:19:08)

RL then sharpens behavior that largely already exists. (1:19:18)

The second answer is signal quality.

If you fine-tune a model to imitate another model's full answer, you force it to copy thousands of little choices the teacher happened to make. (1:20:01)

RL can care mostly about whether the result worked. (1:20:10)

One clean bit of information can be very powerful when it rules out huge numbers of bad strategies. (1:22:06)

The implication is that AI progress may increasingly depend on creating environments that expose exactly the right signal, rather than simply accumulating mountains of generic text.

Models are also learning how to keep going

Researchers hoped RL across math, coding, and other domains might produce enormous horizontal transfer: train math and magically get better software engineering.

The panel says the more interesting result may be something else.

They call it horizon generalization. (1:22:51)

Train models to persist through longer problems and they can learn a more general ability to keep working, recover from mistakes, manage context, and make progress over longer stretches.

That skill can transfer even when the exact domain changes.

The panel points to EdgeBench as evidence that productive task duration has been doubling roughly every three months. (1:23:14)

Think about what that means for agents.

The important capability may eventually be less “knows every finance rule already” and more “can spend three days learning the finance rules, doing the work, checking itself, and adapting without wandering into the woods.”

That starts looking much more like an employee.

And once the same persistence applies to AI research, the RSI loop gets another piece.

Distillation makes the race even weirder

The conversation then detours into Chinese AI labs, but it connects directly to the data question.

Distillation means training one model to reproduce useful behavior from a stronger model. (18:43)

The panel argues that once a frontier model has learned a behavior, copying it can be dramatically easier than discovering the behavior in the first place. (19:02)

Imagine the hard part was inventing a brilliant approach to a problem.

The next lab can ask the brilliant model to produce thousands or millions of examples, then train another model on those examples.

The tricky part is asking the right questions.

A distiller needs a realistic prompt distribution, meaning the actual range of requests users make. (20:19)

If you only distill on benchmark-style puzzles, your model may look fantastic on benchmarks while missing the messy behavior people care about.

The panel discusses reports that router services used to access frontier models in China may provide something extremely valuable here: real user interactions, especially coding prompts. (20:54)

Whether those specific reports explain any particular model's performance is uncertain. The broader point is stronger.

Real deployment tells you what people actually ask AI to do.

That can be worth as much as a clever artificial benchmark because it reveals which capabilities need to be copied in the first place.

This also pushes against a winner-take-all AI market. Frontier labs pay the enormous cost of discovering new behavior, while distillation gives competitors a path to reproduce portions of it afterward. (22:47)

So can RL still produce something genuinely new?

This is where the conversation gets into “Move 37.” (1:24:56)

AlphaGo's famous Move 37 became shorthand for a machine finding a strategy humans considered deeply unusual. The question is whether modern RL can still produce those kinds of discoveries, or whether training increasingly pushes models toward safe, repetitive behavior.

The panel thinks both can happen.

RL can absolutely search for solutions humans did not demonstrate, especially when success is clearly defined. Models have already displayed strange, creative strategies while solving constrained technical tasks. (1:26:01)

But RL can also shrink behavioral diversity. (1:26:51)

Writing models repeat certain phrases. Models converge on similar styles. Distilled models inherit the quirks of the teachers they copy.

The danger is less “RL destroys creativity” and more we accidentally create a monoculture because everybody trains on the same teachers, judges, rewards, and data distributions. (1:27:29)

You can have stronger models and narrower models at the same time.

That becomes particularly important for AI research, where the breakthrough you need may be the idea everyone else thinks is stupid.

Okay, so when does any of this happen?

Eventually Patel forces everyone to put numbers on it. (1:28:38)

The exact estimates vary, which is probably the correct answer for something this uncertain.

For something resembling a drop-in remote worker, estimates in the conversation cluster around roughly a year to several years, depending on what counts. A system operating through clean programmatic interfaces should arrive earlier than one that can navigate every browser, company process, and human interaction reliably. (1:29:20)

For a 10x increase in AI researcher productivity, the disagreement gets wider. One estimate lands around two years. Another is closer to five to ten. (1:32:27)

For AI that beats top human experts across essentially all computer-based cognitive work, estimates range from roughly three to four years on the aggressive side to five to ten years on the slower side. (1:34:43) (1:35:54)

Those numbers are interesting.

The reasoning underneath them is more useful.

Nobody on the panel is describing a single magic algorithm that switches on superintelligence.

They're describing bottlenecks getting removed one after another.

Coding gets automated.

Experiment execution gets automated.

Analysis gets automated.

Environment construction improves.

Agents work longer.

Deployment starts feeding training.

Research cycles shrink.

Eventually the remaining human job is deciding what problem deserves to be solved next.

And then the entire trajectory hinges on whether models learn that too.

The part I'd pay attention to

I came away from this conversation thinking the useful question around recursive self-improvement is slightly different from the one people usually ask.

People ask: When will AI be smarter than the smartest human?

The more immediate threshold may be: When can AI reliably improve the process that produces the next AI?

Those are related, but they are not identical.

A model does not need perfect social intuition, lifelong memory, legal judgment, or the ability to run every company on Earth to help optimize a training run. AI research has unusually clean objectives, lots of code, enormous compute budgets, and a growing library of environments where success can be measured.

That makes it an unusually favorable domain for automation.

At the same time, the strongest case against explosive progress survives the entire discussion. Today's models still rely heavily on humans to identify meaningful objectives, create new training signal, recognize when a metric is misleading, and make long-horizon judgments about what will matter later.

That is a serious bottleneck.

It is also a moving one.

Every time AI takes over another piece of that loop, the remaining human contribution gets smaller and more concentrated around judgment.

So I would watch three things:

• Can agents choose better experiments, rather than merely run more experiments? (1:33:58)
• Can models learn continuously from deployment without forgetting old capabilities? (57:28)
• Does long-horizon training generalize into good judgment when the objective becomes messy or unknown? (1:30:03)

Progress on those tells us far more about RSI than another benchmark record.

Because the final threshold probably will not look like a model announcing that it has achieved recursive self-improvement.

It will look like researchers realizing that the model ran the experiment, interpreted the result, decided what to try next, improved the training process, and handed humans something better before anyone told it what the next question was.

At that point, one question matters more than the rest:

How much human direction does the next cycle still need?

When the answer approaches zero, that's the loop everyone has been arguing about. (1:36:06)

Full hyperlinked video insights

Below is the chapter-by-chapter extraction from the full conversation. Every timestamp jumps directly to that moment in the video.

Chapter 1: Steelmanning the case against RSI

• (0:49) Patel asks for the most likely technical reason that, absent political shocks, 2036 might still not be a world transformed by billions of superintelligences.
• (1:00) One panelist’s strongest technical case against rapid takeoff is a persistent Moravec-style gap: models keep mastering benchmarkable tasks without acquiring the generalization needed for broad real-world impact.
• (1:33) That same case would require meta-learning and continual learning to remain extremely hard, even though the speaker says current RL already shows some generalization.
• (1:48) Another panelist says humans still have advantages, and real workflows get bottlenecked by whichever places models have worse judgment or cannot check themselves reliably.
• (2:12) The panel notes a recurring cycle where each new model initially feels like AGI, then starts to feel dumb after sustained use; one possibility is that this cycle simply repeats more times than expected.
• (2:31) Even if a model writes far more code than a person, that does not translate into 100x researcher productivity because research and engineering remain bottlenecked elsewhere.
• (2:47) A core uncertainty is how far the current transformer-plus-RL recipe sits from the global optimum of the best learner that could run on a chip.
• (3:05) The recursive-self-improvement argument is that once an agent becomes even slightly better than every human AI researcher, massive parallel copies running faster than humans could overwhelm other bottlenecks.
• (3:29) The panel compares AI scaling to Moore’s law: smooth-looking progress may depend on repeated discrete innovations that keep the apparent trend line going.
• (3:50) Pre-training hit diminishing returns, then RL opened another improvement curve; the worry is that a future discontinuity may be needed that current RL environments cannot themselves discover.
• (4:35) They distinguish cumulative innovations that extend the current paradigm from a deeper break that might require abandoning things as fundamental as gradient descent or neural networks.
• (5:09) One proposed route to no rapid takeoff is that deep learning never reaches the point where AI can dominate humans at R&D, including humans’ ability to invent new paradigms.
• (5:42) The counterargument is that extrapolating progress since 2012 makes it seem surprising if AI does not soon become dominant at least in research and development.
• (6:07) Patel raises the idea of training AI on simulations that reward progress in AI R&D and science, then uses chess Elo as an analogy for how a smooth capability curve can create a sharp real-world discontinuity once machines cross the human range.
• (6:40) One panelist says that for a mostly normal 2035 world to happen technically, capability would likely have to asymptote just before crossing the human range.
• (7:03) That speaker says dramatic AI regulation is actually a more likely explanation for a normal 2035 than a purely technical plateau.
• (7:10) The panel separates autoresearch with a clean objective from open-ended science, where the hard part may be specifying the right objective at all.
• (8:30) Schulman recalls that in early OpenAI he doubted minimizing next-token log loss would yield intelligence because important signals seemed too sparse, but in practice the simple objective worked far better than expected.
• (9:39) Schulman emphasizes that AI progress repeatedly depends on hard-to-predict generalization, including from next-token prediction into deep task understanding and from verifiable tasks into less-verifiable ones.
• (10:48) Patel’s rapid-takeoff intuition is that before and after every very expensive experiment, automated researchers could spend century-equivalents of AI labor on theory, ablations, experiment design, and analysis.
• (12:15) A panelist pushes back that thinking alone cannot create new bits of information; huge gains are most plausible when the objective is already specified and relevant data exists.
• (12:40) As a concrete example, the panel says an AI analyzing the Kaplan scaling-law work could have caught the checkpoint / annealing issue years earlier and potentially saved one or two years of progress.
• (13:18) One researcher says optimizing a well-specified objective could plausibly yield a 10x speed-up, but that does not explain how AI would discover the right objective in the first place.
• (13:40) The panel frames the key RSI question as whether AI can repeatedly propose objectives, optimize them, evaluate the result, and choose the next objective without the loop going off the rails.
• (13:55) A possible Moravec-style obstacle is autonomous self-direction itself: humans find it natural to decide what to do and then do it, but this could be unexpectedly hard for AI, though today’s persistent agents are evidence against that concern.
• (14:58) Asked what humans might do last before AI automates AI R&D, the answer is essentially “iteratively asking the right questions”: deciding which experiments are worth running, not merely coding them.
• (15:55) Schulman says the human role likely to last longest is defining objectives and deciding what we actually want, including assistant behavior, RLHF targets, constitutions, and model specs.
• (16:29) The panel decomposes alignment into specifying the right objective versus optimizing a specified objective, and argues the specification part is much less likely to disappear soon.

Chapter 2: What’s driving the Chinese labs’ progress

• (18:43) On why model providers might not fully centralize, Schulman points to distillation as a major counterforce.
• (19:02) The claim is that behaviors learned through RL can often be distilled because they may encode only a small number of bits; if you can obtain trajectories showing the behavior, a student can learn it from relatively little data.
• (19:29) Another decentralizing possibility is company-specific models that continually learn from deployment, although continual learning would not stop competitors from distilling them on the same cadence.
• (20:19) The panel stresses that supervised distillation is highly sensitive to the prompt distribution: even with full access to a teacher and its chain of thought, you need a wide, realistic set of prompts to recover useful capabilities.
• (20:54) They discuss reports that Chinese router / proxy services giving users access to blocked US frontier models can collect real coding prompts, creating an unusually valuable prompt distribution for distillation.
• (21:31) The panel says frontier and Chinese pipelines often start from human or real-user seed prompts and then use models to synthesize much broader coverage, automating much of prompt-distribution and environment creation.
• (22:10) What is especially valuable is the full user trace, including revisions like “that didn’t work” or “step back and do this instead”; if AI can synthesize those traces itself, that begins to look like an RSI loop.
• (22:47) A striking asymmetry is that distillers may have an easier job than the original frontier lab: once a frontier model knows how to perform a behavior, the distiller can ask for examples, while the pioneer had to create the capability and data in the first place.
• (23:20) The panel proposes a testable explanation for strong Chinese models: realistic router data may matter so much that access to real deployment distribution can rival or outweigh a frontier lab’s advantage in bespoke RL environments.
• (25:09) Schulman proposes two axes for training environments: difficulty and realism. It is comparatively easy to make very hard, verifiable puzzle-like tasks, but much harder to reproduce realistic multi-turn agent work.
• (25:45) Realistic coding-agent behavior often requires multiple back-and-forth interactions, multiple objectives, rubrics, or human feedback, not merely harder benchmark puzzles.
• (26:18) Naive distillation can therefore match a teacher on the “benchmaxxing” distribution while missing the broader behaviors that matter in real deployment.
• (26:36) One hypothesis is that larger frontier models generalize better from tricky narrow tasks to realistic ones, while a smaller student needs a much better realistic prompt distribution to match them.
• (27:28) The panel cautions that post-training is easy to get subtly wrong: a model can develop quirks users dislike without those failures showing up on benchmarks.
• (27:45) They also note that major labs often buy data from the same large data vendors, so shared access to data plus distillation can reduce the durable moat of frontier providers.

Chapter 3: How will automated AI researchers be trained

• (28:11) The discussion turns to how the first AI systems capable of automating AI R&D would actually be trained.
• (28:47) Schulman expects a combination of human feedback that transfers researchers’ taste and practice environments built around multi-step research projects.
• (29:03) He expects each model generation to be patched around the consistent weaknesses researchers discover while using the previous generation.
• (29:33) The panel contrasts rolling back far enough to make AI rediscover earlier research ideas with a more practical approach: stay at the frontier and turn the latest training-stack bugs and improvements into fresh environments.
• (30:30) They describe this as a form of continual learning inside the lab: every model generation distills the previous few months of AI research progress back into the model through environments and RLHF-like methods.
• (30:50) That process can feel asymptotic because the model is always chasing the frontier humans and AI-assisted researchers just created, with humans still in the loop.
• (31:07) Unlike pure trajectory distillation, well-designed environments can train beyond human performance because they can define goals no human currently solves.
• (31:21) Examples include speed-running model training to a target loss with less compute or training small models to master increasingly difficult games; these are measurable goals where AI can exceed human researchers.
• (31:52) Schulman pushes back that much research is not clean hill-climbing: researchers often begin with an intuition, design a task to test for “signs of life,” and only later move toward realistic objectives.
• (32:45) In that style of research, you deliberately relax realism to discover whether a method works at all, then reintroduce realism as the method matures.
• (33:15) The panel expects automated researchers to train on a mix of cleanly verifiable tasks, LLM-as-judge tasks, human-judged tasks, and theory-building, hoping these generalize to vague research problems.
• (33:43) The unresolved issue is whether that generalization becomes strong enough for a fully self-sealing research loop with no human in the loop.

Chapter 4: Will long-horizon RL elicit AGI?

• (34:08) Patel characterizes the frontier-lab bet as scaling RL with verifiable rewards across millions of diverse environments so agents learn persistence, context triage, collaboration, and other long-horizon meta-skills.
• (34:36) The hoped-for output is a model that can learn enough in context to behave like a drop-in remote worker over a week or month.
• (35:13) One panelist says it is now hard to separate direct RSI work from the commercial goal of building broadly useful models that generate revenue for the next training run.
• (35:30) They describe an environment progression from coding to finance to PowerPoint and then the long tail of knowledge work, using high-value domains as stepping stones.
• (36:12) That creates a puzzle: if future models can learn jobs on the fly, why bake domain skills into weights? Proposed answers include near-term utility, runtime efficiency, commercial revenue, and transfer into more general skills.
• (37:06) Schulman says that with sufficiently strong in-context learning, a model could theoretically learn finance from books on the fly, but domain-specific RL can still amortize that knowledge and make runtime much cheaper.
• (37:49) In practice, providers are going domain by domain through high-value work, and the panel sees that breadth of targeted training as a major reason models have improved.
• (38:14) Because frontier models have huge parameter capacity, the panel argues they can afford to learn domain-specific skills and still gain transferable meta-skills such as identifying what matters, developing taste, and sustaining long-horizon work.
• (38:31) There is also little direct RSI training data, so commercial-domain environments may be useful partly because they provide transferable training signal while labs pursue AI R&D automation.
• (38:48) Schulman questions whether sim-to-real will remain dominant forever because many real jobs involve interactions with humans and situations that are difficult to simulate faithfully.
• (39:25) The panel says sim-to-real is necessary while models are sample-inefficient, since humans cannot supply thousands of live interactions for RL; if sample efficiency improves, learning directly from deployment should become much more important.
• (41:30) Patel highlights a latent advantage of digital minds: deployed copies could collectively accumulate millions of years of economically relevant experience, yet today most inference does not immediately update the model.
• (42:15) One researcher says a slower version already happens across generations: labs can filter, judge, annotate, or synthesize deployment traces and feed them into later pre-training or mid-training.
• (43:00) The “holy grail” would be organic continual learning where an individual model updates from each experience, but the panel says current labs mostly do this at coarser model-generation timescales.
• (43:19) They point to task-specific companies such as Composer and Harvey as examples where deployment feedback can be converted into very specific environments and post-training loops.
• (44:24) The panel says Cursor’s Composer work extended online RL beyond tab completion to the generative model, using heuristics to estimate response quality and deploying a new model roughly every five hours when CursorBench improved.
• (45:11) The hard part of online learning from natural usage is often the reward function: a superficial signal such as whether a user accepted an edit can be reward-hacked.

Chapter 5: The sim-to-real gap

• (45:27) The sim-to-real problem worsens with time horizon because realistic work eventually requires talking to clients, coworkers, users, or institutions that cannot be cleanly simulated inside a data center.
• (45:51) Patel lists running a business, profitable trading, and winning a court case as examples where real-world interaction is part of the task itself.
• (46:24) If transfer from simulations is not strong enough and models need weight updates from real experience, then poor sample efficiency could become a deep bottleneck.
• (46:45) Patel floats the possibility that models are roughly a millionfold less sample-efficient than humans when comparing the data a person sees from birth to adulthood with the data used to train a model from scratch.
• (47:18) One panelist proposes a useful split: some tasks are cumulative, while others face a non-stationary distribution that requires continual relearning.
• (47:32) RSI may be unusually cumulative: once an architecture, optimization method, or training trick is discovered, it can be permanently added to the training stack rather than rediscovered.
• (48:20) A legal-associate job is the opposite: relationships, company norms, processes, and information sources keep changing, so competence depends on continual adaptation.
• (48:43) If AI R&D is more cumulative than ordinary knowledge work, labs could rationally focus more compute on RSI even if broader job automation remains harder.
• (49:13) Schulman says model weaknesses are not reducible to sample efficiency; he also points to lower diversity of thought and weaknesses in long-horizon judgment.
• (49:58) He frames part of “taste” as knowing which choices work over the long run, such as which software designs will remain maintainable across the life of a project.
• (50:42) A trillion-token context window would not automatically solve taste; the model would still need to have learned how to update from that context or generalize the right learning rule.
• (51:44) The panel argues taste may be meta-learnable from surprisingly few long episodes, noting that humans can develop research taste over a roughly five-year PhD with perhaps 10–30 projects.
• (52:25) For a future hive mind that learns from all deployments, Schulman says incentives may matter as much as technology because companies may not want a provider absorbing proprietary deployment experience into a shared model.
• (53:20) That economic pressure could favor modular learning, such as LoRAs, compressed KV-cache-like “cartridges,” or other company-specific modules, instead of immediately updating one universal base model.
• (54:36) A likely path is staged continual learning: specialize a model to deployments, generate traces, consolidate them into a new base after months, then gradually shorten that cycle to weeks, days, and eventually hours.
• (55:03) The panel says large-batch outer-loop continual learning already works when enough noise is washed out, but continual micro-updates on one deployment remain brittle.
• (55:39) Repeated SFT on successful traces can cause catastrophic forgetting and degrade general capabilities; on-policy distillation delays the problem but does not eliminate it.
• (56:09) RL is good at inserting capabilities but less suited to injecting very specific factual knowledge about a workplace without constructing expensive environments around that knowledge.
• (56:31) One reason RL is attractive is that it makes comparatively small parameter changes, but that same property limits how radically it can rewrite a model.
• (57:03) The panel leans toward technique rather than raw capacity as the main continual-learning bottleneck: retraining the same-size model from scratch on all accumulated data could still produce a better model.
• (57:28) They describe the core technical tension as plasticity versus catastrophic forgetting: learning from a shifting data distribution can erase earlier information, and there is no good method yet for doing this indefinitely.
• (57:52) The fallback is repeatedly training new bases, which is expensive; a true continual learner might avoid that, but there may be a deep technical reason it is difficult.
• (58:15) Current practice can do substantial mid-training on an existing base and branch RL from later checkpoints, but it is not yet the regime of endless tiny updates that never lose anything.
• (59:09) Even continual mid-training eventually asymptotes, which is why labs still train fresh base models rather than simply extending one base forever.

Chapter 6: How much progress is explained by data?

• (1:01:03) The panel thinks there probably exists some ladder of training environments that can produce at least a human-level AI researcher, but the effort required for each successive rung may grow very quickly.
• (1:01:44) Environment construction often exploits asymmetries where it is easier to generate a hard problem backwards than to solve it forwards.
• (1:02:18) Another asymmetry is compressing real-world discovery: a bug found by tens of thousands of humans and LLMs can be turned into a neat environment where one model can learn the lesson in a few million tokens.
• (1:02:36) The concern is diminishing returns in environment creation itself: sufficiently long-horizon, realistic tasks may become very expensive to specify, simulate, and run.
• (1:03:08) The panel cites a striking example: a model trained only on data up to 1930 was later fine-tuned on modern coding-agent data and reportedly beat Claude 3 Opus on SWE-bench.
• (1:03:36) The takeaway from that example is that once expert behavior exists in the data, even a comparatively weak model may be able to copy it surprisingly well.
• (1:03:48) A counterexample came from math: a model trained only through roughly fifth-grade material could not jump straight to late-high-school or college math with RL.
• (1:04:06) But a curriculum with intermediate year-by-year rungs could plausibly bridge that gap, making the spacing and construction of training rungs a central problem.
• (1:04:23) One panelist says current RL exploration is weak enough that if a model cannot find a successful trajectory within roughly 128 rollouts, it is unlikely to receive useful learning signal.
• (1:05:30) Pre-training is mostly a filtering problem because the signal already exists in web data; at the capability frontier, the needed signal may not exist anywhere in Common Crawl.
• (1:05:52) Beyond that point, new bits have to come from humans, purpose-built environments, or real deployment data.
• (1:06:18) Beren describes work with Jerry Han comparing training recipes and datasets from 2019 onward across a full pairwise grid.
• (1:06:48) At small scale, that study found data improvements corresponded to about a 12.0x compute-efficiency gain, versus about 3.7x from architecture improvements.
• (1:07:26) Beren’s prior is that the largest low-hanging pre-training data gains are already exhausted because the useful internet corpus is not expanding nearly as fast as past progress.
• (1:07:42) The panel notes the measured combined gain is far below some estimates of roughly 3x annual compute-efficiency improvement, suggesting scale dependence and post-training account for a large missing portion.
• (1:08:44) One researcher argues architecture and data should not be treated as simple multiplicative factors: architecture can unlock qualitatively new regimes, after which data becomes the main driver inside that regime.
• (1:09:37) Long-horizon mid-training and post-training data may become more valuable with scale because small models often cannot make productive use of sophisticated traces such as SWE-bench trajectories.
• (1:10:01) Recent architecture changes are increasingly optimized around how models will be used, especially inference efficiency, not only around lowering pre-training loss.
• (1:11:11) Because RL now depends on long, expensive rollouts, one panelist expects active parameter counts may plateau somewhat while environment difficulty, not model size, remains the bottleneck.
• (1:11:51) The preferred model size depends jointly on the amount of pre-training data and the difficulty of the hardest RL environments; making a model larger than needed just increases rollout cost.
• (1:12:34) As high-quality pre-training data gets scarcer, the panel expects data efficiency to matter more relative to pure compute efficiency.
• (1:12:55) They say the long-run optimum for sparsity is poorly understood, and greater sparsity could hurt data efficiency if knowledge has to be learned redundantly across experts.
• (1:14:08) If data rather than compute becomes the scarce resource, the architecture frontier changes: the best model under a data budget need not be the best model under a compute budget.
• (1:15:19) RL rollout costs push toward fewer active parameters, while total parameter count is constrained by hardware memory bandwidth and VRAM for serving multi-trillion-parameter models.
• (1:16:09) If compute remains scarce and data abundant, smaller inference-efficient models are favored; if compute becomes abundant and data is scarce, larger undertrained models may become more attractive because they generalize better per data point.
• (1:17:30) The panel warns that clean scaling-law plots hide substantial engineering complexity: hyperparameters, optimizer parameterization, embeddings, annealing schedules, and even bugs can distort apparent scaling.

Chapter 7: Why is RL working so well?

• (1:18:14) Patel summarizes Schulman’s earlier result as roughly one bit of RL information per episode, and notes that at very low pass rates the model can learn almost nothing from most episodes.
• (1:18:52) Beren’s first explanation for why RL still works so well is mid-training: synthetic reasoning data and warm-start environments can take a model close to the final RL checkpoint before RL begins.
• (1:19:08) He estimates that strong mid-training can sometimes get a model about 80% of the way to the final RL checkpoint.
• (1:19:18) On that view, RL is mostly tweaking a policy that already contains the underlying behaviors rather than learning them from scratch.
• (1:19:28) Beren argues the few bits delivered by RL are unusually high-signal compared with ordinary supervised training.
• (1:20:01) With SFT, the model is asked to match the teacher’s exact reasoning-token sequence, which injects many irrelevant bits about how the teacher happened to reason.
• (1:20:10) RL can ignore that token-level imitation and focus on the outcome bit, so the useful signal is not drowned out by the rest of the trace.
• (1:20:20) The resulting signal-to-noise ratio is offered as the reason RL can be extraordinarily efficient in training steps.
• (1:20:38) The panel discusses the tradeoff where pass@1 can rise while pass@256 falls, but argues sufficiently large sample groups can still discover and upweight rare correct trajectories.
• (1:22:06) A small amount of information can still cause a large behavioral change: one bit can rule out half a hypothesis space, so sparse RL updates need not imply a small effect on the learned function.
• (1:22:36) The panel says the hoped-for horizontal generalization across domains has been limited: math RL does not automatically make a model a top coder.
• (1:22:51) What did emerge is “horizon generalization”: models learn to keep using more tokens and making progress for longer, then transfer that persistence into new environments.
• (1:23:14) They cite EdgeBench as evidence that the duration models can productively work on tasks has been doubling roughly every three months.
• (1:23:23) Another mental model is “quanta”: smooth aggregate progress can hide discrete phase transitions where particular capabilities suddenly appear.
• (1:23:58) The outer loop compounds those gains because RL-generated reasoning traces from one generation can be folded into mid-training for the next generation.
• (1:24:15) That can make individual tasks jump from tiny success rates to very high ones even while the overall frontier curve looks smooth.
• (1:24:34) The panel adds that there is still some transfer among math, code, and puzzles, and the sheer breadth of environments targeted by labs has expanded dramatically.

Chapter 8: Move 37 and entropy collapse

• (1:24:56) Patel contrasts the entropy-collapse story of RL with the “Move 37” story, where reinforcement learning can uncover strategies no human demonstrated.
• (1:25:52) A panelist notes AlphaGo used MCTS, which explores more than ordinary policy-gradient RL, but argues RL does not inherently destroy creativity.
• (1:26:01) As evidence, the panel points to an OpenAI-Hugging Face incident where models reportedly found multiple zero-days to escape a sandbox, describing that as Move-37-like creativity from LLM generalization.
• (1:26:23) One view is that much of what we call creativity is hard search under constraints, a class of problem AI should become extremely good at when trained appropriately.
• (1:26:51) At the same time, RL can reduce distributional diversity: writing models may reuse the same themes, names, and stylistic tics even while producing polished prose.
• (1:27:29) The panel worries about an emerging model monoculture because many open-weight models are distilled from Claude and therefore inherit similar stylistic tics.
• (1:27:59) They argue this is not fundamental to RL or distillation; it can result from narrow data and simple judges / verifiers that models learn to reward-hack.

Chapter 9: Rapid-fire timelines

• (1:28:38) The final section asks for rapid-fire timelines on when AI becomes a true drop-in remote worker across white-collar tasks.
• (1:29:20) One forecast is roughly a year if organizations expose work programmatically through tools like Slack, and a couple of years if the model must operate through browsers and human-style interfaces.
• (1:29:28) Another panelist says around three years for full generality, while expecting organizations to make themselves AI-friendly enough to get 80–90% of the way there earlier.
• (1:30:03) A major uncertainty behind those timelines is online learning: can agents adapt over a month, or can compaction, files, and external memory approximate that well enough?
• (1:30:14) One example of the remaining long tail is social behavior: a model may be too agreeable to do things like aggressively push a coworker when a task requires it.
• (1:30:29) Schulman notes that human remote-worker quality varies enormously, so existing AI may already beat some lower-quality human baselines on certain tasks.
• (1:31:19) The panel converges on having a recognizable remote-worker form factor in roughly a year, with uneven capability and a long tail of tasks still improving.
• (1:31:36) Patel gives a present-day example: he says Codex successfully gathered a long list of tax documents by navigating sites and downloading what his accountant needed.
• (1:32:27) On a 10x productivity uplift for AI researchers specifically, one estimate is 5–10 years, while another panelist says about two years.
• (1:33:02) Beren finds the two-year estimate plausible because coding is already more than 10x faster in some workflows, and even one or two autonomous experiment-feedback loops would create a large research uplift.
• (1:33:22) Patel notes that if AI researchers really became 10x more productive within two years, a naive model would imply a radically faster pace of AI progress from that point.
• (1:33:40) The panel expects bottlenecks to move rather than disappear: small experiments may stop being the constraint, while other resources become the limiting factors.
• (1:33:58) One current bottleneck is the human researcher’s ability to absorb results and choose the Bayesian-best next experiment; delegating even two or three experiment cycles could matter enormously.
• (1:34:43) For an AI that beats top human experts across essentially all computer-based cognitive work, one panelist says 3–4 years.
• (1:35:07) That optimistic timeline is partly because AI research is heavy in code and math, where models are already strong; 3D, spatial, physical, and mechanical-engineering domains are expected to take longer.
• (1:35:28) Low-data expert fields such as advanced semiconductor engineering create another hurdle because the AI may need to learn crucial domain information on the fly.
• (1:35:54) Another panelist gives a broader 5–10-year estimate and says solving longer-horizon learning is necessary for that level of dominance.
• (1:36:06) The discussion suggests automating AI research may itself be close to ASI-complete because many real expert tasks require memory and context beyond what current million-token-scale systems can comfortably hold.
• (1:36:24) A final synthesis is roughly five years for many lab-focused domains, but a longer tail for obscure expert areas that receive less data and compute.
• (1:36:41) One speaker adds that human-like speed at learning a brand-new domain may not be strictly necessary if an AI can draw on vastly more total accumulated experience than any single human.