The dominant architecture in Large Language Models (LLMs) today faces a fundamental bottleneck: the autoregressive process. Whether it is ChatGPT or Claude, these models generate text sequentially, one token at a time. It is a linear process that is inherently difficult to accelerate, regardless of how many GPUs you throw at the problem.
Stefano Ermon, founder and CEO of Inception Labs, believes the industry has hit a wall with this approach. His solution? Mercury, a new commercial-scale language model built on diffusion technology rather than autoregression.
"Everybody's building a so-called autoregressive model which essentially works by generating text or code one token at a time... that's pretty slow," Ermon told The Neuron at AWS re:Invent. "We're building a completely different kind of LLM that is based on a diffusion model."
From Noise to Refined Code
Diffusion models are already famous in the image generation world (powering tools like Stable Diffusion and Midjourney). They work by starting with pure noise and iteratively refining it into a clear image. Inception Labs has successfully applied this same logic to text and code generation.
Instead of predicting the next word in a sentence, the Mercury model processes the entire output simultaneously.
"Instead of generating an answer one word at a time, we start with noise. We start with a guess... and then we gradually refine it until we get the correct answer," Ermon explains.
This architecture allows for massive parallelization. Because the model is not waiting for "Token A" to exist before creating "Token B," it can utilize GPU compute power much more efficiently. Ermon notes that in a diffusion language model, "each neural network evaluation is able to output or modify many tokens. So for the same cost, you get more things out."
The Pareto Dominance of Diffusion
The implications for enterprise infrastructure are massive. Inception Labs claims their Mercury model is up to 10 times faster than comparable autoregressive models. Here is a recent paper from Inception detailing the levels of speed his team is reaching.
"For the first time, a diffusion language model was competitive with a small autoregressive model... and it was like 10x faster," Ermon says, referencing their initial research at Stanford. Now, operating at commercial scale, the efficiency gains remain drastic.
"We're basically Pareto dominating autoregressive models in terms of throughput versus latency," Ermon adds. This means for any given speed requirement, Mercury is cheaper; for any given cost constraint, Mercury is faster.
Built-in Error Correction
Beyond speed, the diffusion process introduces a unique mechanism for accuracy: self-correction. Since the model iteratively refines "noise" into "text," it effectively edits its own work during the generation process.
"A diffusion model is actually trained to correct mistakes and during inference when it generates an answer, it keeps refining, it keeps fixing mistakes," says Ermon. Unlike an autoregressive model, which cannot "take back" a token once generated, Mercury is constantly optimizing the entire output until completion.
Commercial Availability and Backing
The industry is taking notice. Inception Labs recently closed a $50 million seed round backed by early investors in OpenAI and Anthropic, alongside strategics like Microsoft, Nvidia, Databricks, and Snowflake.
The Mercury model is currently available via an OpenAI-compatible API and AWS Bedrock, making it a drop-in replacement for low-latency applications like coding assistants and voice agents.
As the demand for AI inference grows, the shift from sequential to parallel processing may be the software breakthrough that prevents a hardware bottleneck. "We believe that this is going to be the future of language models," Ermon concludes.
Inception Labs Q&A
Corey Noles: Can you tell us a little bit about Inception Labs and what you guys are doing over there?
Stefano Ermon: At Inception, we're building a new kind of LLM that is based on diffusion. Everybody's building a so-called autoregressive model which essentially works by generating text or code one token at a time, and that's pretty slow. It's very hard to accelerate through parallel compute. We're building a completely different kind of LLM. You might have seen models that can generate images or video—those models don't generate an image one pixel at a time. They generate it through a so-called diffusion process. We’ve taken that idea and figured out how to apply it to code generation. Instead of generating an answer one word at a time, we start with noise. We start with a guess, and then we gradually refine it until we get the correct answer.
Corey Noles: Tell me about the decision to go that route instead of traditional text direction.
Stefano Ermon: In my lab at Stanford back in 2019, we invented this area using a diffusion model. Since then, I've been trying to get diffusion models to work on text and code because they are so much faster. Diffusion models can generate things instantly. We published a paper last year showing that for the first time, a diffusion language model was competitive with a small autoregressive model, and it was like 10x faster. So I took that idea and started Inception to scale up.
Corey Noles: Do you feel like it brings an extra level of accuracy by the approach it takes?
Stefano Ermon: That's another benefit of a diffusion model. If we think about an autoregressive model, once it outputs something, it can never take it back. A diffusion model is actually trained to correct mistakes, and during inference when it generates an answer, it keeps refining, it keeps fixing mistakes. We are pretty optimistic that this kind of error correction is going to allow us to achieve higher accuracy compared to autoregressive models.
Corey Noles: Ideally you'd be able to get more out of the same GPU?
Stefano Ermon: Yes, absolutely. We can either be faster than autoregressive models for the same cost or, for the same speed, we can actually be 5 to 10x cheaper. We're basically Pareto dominating autoregressive models in terms of throughput versus latency.
Corey Noles: I believe you all closed a $50 million seed round recently?
Stefano Ermon: That's right. We closed a $50 million seed round with some top investors including early backers of OpenAI and Anthropic, and strategics like Microsoft, Nvidia, Databricks, and Snowflake. We have a really amazing group of supporters that believe that this is going to be the future of language models.
Corey Noles: If someone wants to go and try Mercury, what's the best way to do that?
Stefano Ermon: Mercury models are available through our API platform. It's all OpenAI compatible. So whatever app you're building on traditional autoregressive models, you can replace them and use the Mercury diffusion language model. We're also on Bedrock. We partnered closely with AWS, so you can actually deploy Mercury models inside your VPC.
