AI Safety’s New Paradox: America May Need a Bigger Lead to Slow Down

AI Safety’s New Paradox: America May Need a Bigger Lead to Slow Down

President Donald Trump and Anthropic CEO Dario Amodei agree on one surprisingly important thing:

The United States cannot afford to lose the AI race to China.

Their disagreement starts with what America should do with that lead.

Trump treats U.S. advantage as something to press. Asked this weekend whether AI companies should slow development or face more regulation, he pointed straight to China.

“We’re leading China in AI,” Trump said. “Whoever wins AI wins.”

Amodei also thinks a Chinese AI lead would be dangerous. But his newly proposed framework reaches a very different conclusion: America needs enough of a technological advantage that its leading AI companies can afford to slow capability development when safety falls behind.

In other words, the emerging fight over frontier AI is getting stranger than “speed versus safety.”

Both sides want America ahead.

One wants to keep converting that lead into faster progress. The other increasingly sees part of the lead as something that can be spent buying time.

And that creates a much harder question for AI policy:

How far ahead does America need to be before anyone feels safe enough to slow down?

The safety case now depends on the race it wants to slow

This debate exploded after former OpenAI and Anthropic researcher Jacob Coxon quit Anthropic and warned that frontier labs are moving toward increasingly autonomous AI systems before anyone knows how to reliably control much more capable versions of them.

Coxon’s warning is dramatic. His concern centers partly on recursive self-improvement: AI becoming good enough at AI research that it meaningfully accelerates development of the systems that come after it.

That process does not require a sci-fi machine suddenly rewriting itself.

AI can write research code, run experiments, analyze results, propose follow-up work and handle more of the loop humans use to develop the next model. OpenAI says its own agents are already taking over larger pieces of that research process, something we examined recently at The Neuron.

Coxon’s argument is that companies cannot reliably solve this problem through individual restraint because each company operates under pressure from everyone else.

If Anthropic pauses while OpenAI keeps improving its models, Anthropic loses ground.

If the major American labs coordinate but Chinese labs continue accelerating, the U.S. could lose ground.

So even executives who genuinely believe development is moving too fast have an obvious reason to keep moving.

That incentive problem is the most important part of Coxon’s warning. You do not need to accept his forecast of human extinction to see it.

And Amodei is now trying to turn that problem into an actual governance framework.

Amodei’s plan contains a pretty big catch

Amodei’s proposal has three broad layers.

First, he wants permanent independent evaluators embedded inside frontier AI companies with deep enough access to assess what models can do and whether safety controls are keeping up.

Second, he wants leading AI companies in democratic countries to coordinate around shared safety standards and capability thresholds, potentially with government help resolving antitrust barriers.

Third, he wants international agreements around at least some dangerous AI capabilities.

But buried inside that framework is the constraint that changes the whole story:

Amodei says democracies can only slow down by roughly as much as their lead over geopolitical rivals allows.

If the U.S. slows beyond that margin while China keeps advancing, he argues, China could pull ahead. His proposed response is actually to protect and widen the American lead first through tougher restrictions on advanced chips, unauthorized model distillation and theft of model weights.

In Amodei’s words, a larger lead gives democracies the “breathing room we need in order to pace effectively.”

That flips one common assumption about AI safety on its head.

U.S. technological dominance and slower AI development are usually presented as opposing goals.

Under Amodei’s framework, dominance is partly what makes slower development possible.

Chip controls therefore become more than economic or national-security policy. Within this framework, they become part of the safety architecture: keep rivals far enough behind that frontier labs have room to stop at safety checkpoints without immediately surrendering the frontier.

That is a much more complicated proposition than simply telling Silicon Valley to pump the brakes.

Trump isn’t arguing for zero guardrails either

Trump’s position is also more nuanced than “AI safety bad.”

When reporters asked him about slowing AI development, he acknowledged that “we could put guardrails,” while dismissing some of the more catastrophic warnings as things that “won’t happen.” His priority was preserving the American lead over China.

His administration’s actual policy reflects that balance.

The White House’s AI Action Plan emphasizes faster innovation, infrastructure expansion and U.S. technological leadership.

But a June executive order also directed the government to build classified benchmarks for advanced cyber capabilities in frontier models and create a voluntary framework through which developers can give the government secure access to certain frontier systems for up to 30 days before releasing them to other trusted partners.

The same order explicitly says this system does not authorize mandatory government licensing, preclearance or permitting for AI models.

That distinction matters.

The emerging disagreement is less about whether frontier AI deserves any oversight.

It is about how much power that oversight should have.

The Trump framework can inspect, benchmark and collaborate with companies while deliberately avoiding a mandatory government gate on model releases.

Amodei is contemplating safety checkpoints that could actually constrain how quickly capabilities advance when safeguards fail to keep pace.

That is a much more consequential dividing line than “regulation versus innovation.”

It is the difference between watching the frontier and having a mechanism capable of telling it to wait.

California is testing another version of accountability

California adds another layer.

The state’s SB 53 does not order AI labs to slow model development. Instead, it imposes binding transparency and accountability requirements on large frontier developers.

Companies covered by the law must publish safety frameworks, while the state created mechanisms for reporting critical safety incidents and protecting whistleblowers who disclose significant health and safety risks. Noncompliance can trigger civil penalties.

That makes California useful as a governance experiment, but not because it has installed an AI speed limit.

The bigger question is whether transparency, reporting and whistleblower protection are enough if the underlying problem eventually becomes how fast capabilities themselves are advancing.

Amodei’s proposal suggests they may be different layers of the same system: visibility tells outsiders what is happening; pacing mechanisms determine what happens when the answer looks dangerous.

Wall Street just showed why this gets difficult fast

The competitive pressure is not limited to rival AI labs and governments.

Investors are also pricing enormous expectations around continued AI progress.

After leading AI executives publicly backed slower frontier development, several AI- and semiconductor-linked stocks fell sharply in Asia. SoftBank, SK Hynix, Samsung and multiple Chinese AI names were among those hit.

One day of market trading does not prove that slower frontier development would collapse AI spending.

It does show how sensitive investors are to anything that changes assumptions about the pace of AI progress.

That matters because any serious pacing regime operates inside an industry where capability improvements support valuations, fundraising, product road maps and competitive positioning.

A lab deciding to slow a model therefore is not making a safety decision in isolation.

It is changing expectations throughout an economic ecosystem that has spent years betting on acceleration.

Then there’s China

The geopolitical piece makes everything harder.

After Amodei called for pacing AI development while tightening restrictions intended to preserve the U.S. lead, China’s state-backed Global Times attacked the proposal as a “Cold War” strategy.

The criticism exposes another paradox in the safety plan.

Amodei ultimately wants cooperation with China on issues such as testing dangerous models, restricting biological-weapons uses and potentially limiting extremely rapid recursive self-improvement.

But he also argues that democracies should first protect their bargaining position by keeping their AI lead as large as possible.

From Beijing’s perspective, that can look less like neutral safety policy and more like Washington asking China to cooperate under rules written after America secures the advantage.

This is why global AI coordination gets complicated almost immediately.

The countries involved would have to agree on what capabilities trigger restrictions.

They would need ways to inspect or test models.

They would need confidence that competitors are not secretly training more capable systems.

They would need consequences for cheating.

And they would have to distinguish measures genuinely designed to reduce catastrophic risk from measures designed to preserve military or economic power.

That last part might be the hardest.

AI capability itself is increasingly becoming a strategic resource. As we’ve written before, the competition now covers chips, model weights, APIs, cloud access and even who gets to interact with the most advanced systems.

Safety negotiations are taking place inside that rivalry, not outside it.

So what is America’s AI lead actually for?

This is where the Trump-Amodei disagreement gets genuinely interesting.

Trump’s governing logic is straightforward: America has the lead, so America should protect it by continuing to move faster than China.

Amodei accepts the need for that lead but assigns it another purpose.

A lead creates margin.

And margin creates options.

If American labs are only barely ahead, slowing down feels strategically dangerous. If they are comfortably ahead, a temporary slowdown for testing, alignment work or security improvements becomes easier to tolerate.

That turns the size of the U.S. advantage into something close to an AI safety budget.

Spend too much of it slowing development and a competitor catches up.

Spend none of it and safety work may never get the time its advocates say it needs.

Nobody knows the correct balance because nobody knows the probabilities at the center of this argument.

Coxon’s prediction that advanced AI could eventually threaten humanity remains a warning, not an established forecast. The most extreme loss-of-control scenarios are contested.

But the underlying governance problem already exists.

Models are becoming more autonomous. AI systems are contributing more heavily to AI research. Cyber capabilities are rising. Companies compete with one another. Countries compete with one another. Investors reward continued progress.

So the most important question emerging from this week’s AI panic may not be whether America should race or brake.

It may be how much of its lead America is willing to spend on caution—and how anyone will know when that lead is finally large enough to afford it.