The State of AI in November 2025: A Deep Dive with the Co-Founders of Artificial Analysis

A few weeks ago, we sat down with Micah Hill Smith and George Cameron, the co-founders of Artificial Analysis, for a nearly two-hour conversation about the state of AI. If you've ever wondered how to actually compare AI models, why benchmarks matter, or which model is worth your money—this is the conversation you need.

Fair warning: this is a long one. But if you're serious about understanding AI in 2025, it's worth every minute.

Let's dig in.

Table of Contents

1. Who Is Artificial Analysis?
2. The Wild Week That Changed Everything
3. Google Finally Has the Best Model (Yes, Really)
4. Is AI Progress Stalling? The "Scaling Wall" Debate
5. Token Efficiency: Why Anthropic Models Feel Different
6. The Thousand-X Cost Difference
7. AI Omniscience: The New Hallucination Benchmark
8. Hallucination Is an Incentive Problem
9. Which Models Know the Most About Programming?
10. The Value of Independent Benchmarking
11. Image Generation: The New Leaderboard
12. Video Generation: Where Things Get Interesting
13. Speech-to-Speech: The New Frontier
14. What Models Are Enterprises Actually Using?
15. Why Qwen Dominates Fine-Tuning
16. Choosing a Provider: Speed vs. Cost
17. What About World Models?
18. What Models Do the Experts Use?
19. The METR Task Doubling Benchmark
20. What Would a Perfect Score Look Like?
21. Can AI Actually Do Our Jobs Yet?
22. Key Takeaways

Top Takeaways From the Episode

• (01:38) The "Wild" Week of December 2025: The industry experienced a massive release cycle in just eight days: Google released Gemini 3 Pro, followed immediately by OpenAI's GPT 5.1 Max (aiming for the coding crown), and then Anthropic's Claude Opus 4.5 ("Anthropic Strikes Back").
• (04:42) The Origin of Artificial Analysis: The founders (ex-Google interns) started the project because developers faced a 1,000x price difference between AI models. They needed independent benchmarking to navigate the trade-offs between intelligence, speed, and cost, which was impossible to do with provider-supplied data alone.
• (10:57) The "Tight" Intelligence Gap: When looking at the top 5 to 7 frontier models, there is less than a 10% difference in pure intelligence scores. However, the price difference remains massive (up to 1,000x), making model selection a financial decision rather than just a capability one.
• (12:18) Google's First True Lead: Gemini 3 Pro marked the first time Google clearly held the top spot on the "Frontier Model Intelligence" chart. Prior to Gemini 2.5 Pro, Google struggled to produce a model competitive with OpenAI or Anthropic for nearly two years.
• (15:05) The GPT-5.1 "Router" Confusion: The discourse around GPT-5.1 stemmed from the massive difference in user experience between "5.1 non-reasoning" and "5.1 high-reasoning." The non-reasoning version lacks agentic search tools, creating a confusing product experience where the model feels vastly different depending on the mode.
• (16:05) Rapid Evolution of Coding Agents: Just over a year ago (pre-Claude 3.5 Sonnet), coding agents that could execute multi-step function calls or edit multiple files fundamentally did not work. In roughly 14 months, this capability has gone from non-functional to industry-standard.
• (19:00) Debunking the "Scaling Wall": Micah argues that suggestions of stalled progress are "greatly exaggerated." While progress looks incremental week-to-week, the year-over-year jump (e.g., GPT-4 to o1) was double or more in capability. The base case is continued progress without needing new "miracle" breakthroughs.
• (20:06) Market Saturation: The "Frontier" chart now tracks 13 different companies significantly competing for the top spots, rather than just the expected 3 or 4 (OpenAI, Google, Anthropic).
• (23:48) Microsoft's Hardware Play: Microsoft has built a new data center featuring the largest concentration of Nvidia GB300s. For context, a single GB300 is roughly equivalent to 11 H100s, signaling a massive leap in compute density.
• (24:55) Meta's Strategic Shift: Meta has been quiet since the "Llama Maverick" release in early 2025. The Llama 4 launch was considered "less successful," leading to speculation about whether they will continue with open weights or pivot to first-party products.
• (25:57) The Rise of Small Models: Chinese labs like Alibaba (Qwen) have released models small enough to run on phones (e.g., Qwen 3 4B) that punch significantly above their weight class, rivaling desktop-class models from previous generations.

Technical Deep Dives: Tokens, Costs, & Hallucinations

• (28:57) Token Efficiency Convergence: The distinction between "reasoning" and "non-reasoning" token usage is blurring. Previously, reasoning models used 10x the tokens. Now, some inefficient non-reasoning models use more tokens than efficient reasoning models.
• (31:56) Anthropic's "Vibe" Advantage: Anthropic's models (Haiku, Sonnet, Opus) use roughly 3x fewer tokens in reasoning mode compared to competitors like Kimi K2 or Grok 4. This conciseness drives their popularity among developers—they are cheaper and faster because they output less "fluff."
• (36:38) The $6 vs. $3,000 Benchmark: The cost disparity is staggering. Running the full "Artificial Analysis Intelligence Index" (10 evals) on GPT-5 Nano costs $6. Running the same suite on Claude 4.1 Opus costs over $3,000.
• (39:05) The "Golden Quadrant" Models: The models currently sitting in the high-intelligence/low-cost sweet spot are Grok 4.1 Fast, DeepSeek V3.2, MiniMax M2, and GPT-5s 12B.
• (41:39) The Prius vs. Ferrari Fallacy: Human brains struggle to comprehend the scale of AI cost differences. It's not like comparing a Prius to a Ferrari; it is mathematically like comparing a Prius to 100 Ferraris (or a Ferrari to a Lime Scooter).
• (45:00) AI Omniscience & Negative Scoring: To combat guessing, Artificial Analysis introduced a new "Omniscience" benchmark where models get +1 point for a correct answer and -1 point for an incorrect answer. This forces a measure of reliability rather than just accuracy.
• (48:35) Hallucination is an Incentive Problem: Models are currently trained to guess because standard benchmarks reward a 25% chance of being right over a "I don't know" (0 points). Labs breed hallucination into models to chase high leaderboard scores.
• (53:19) The "Smart Liar" Phenomenon: Intelligence improvements do not correlate with reduced hallucinations. Gemini 3 Pro is significantly smarter than Gemini 2.5 Flash, yet they share roughly the same hallucination rate on hard questions (~88% incorrect when they don't know the answer).
• (57:47) Python Bias in Coding Models = Good?: In software engineering benchmarks, Anthropic and Google dominate Python scores specifically. This suggests massive over-optimization for Python in training, which explains why engineers prefer these models—they "know" the libraries better.
• (59:43) The Swift Anomaly: While Opus leads in almost every language, Gemini dominates Swift programming. This has led to speculation about a strategic data-sharing deal or partnership between Google and Apple.

Future Outlook & Specialized Models

• (01:02:00) The "Apple Battery" Effect of Benchmarks: Internal company benchmarks aren't necessarily "nefarious," but they are curated like Apple's battery life claims. They test under optimal conditions (specific prompts, best checkpoints) that almost never match real-world developer usage.
• (01:07:27) Gemini 3's Secret Weapon (Nano Banana): Google's Gemini 3 Pro (internally codenamed "Nano Banana Pro") has taken the top spot in both text-to-image and image editing, solving the "text on glass" rendering issue that plagued previous models.
• (01:13:45) Video Generation Pivot: The frontier of video AI has shifted from text-to-video to image-to-video. This allows creators to generate a perfect starting frame using an image model and then animate it, offering far more control than random text generation.
• (01:20:08) DeepSeek's Explosion: In the 2025 developer survey, DeepSeek usage went from effectively 0% to 53% in a single year. XAI's Grok saw a similar rise from zero to major player status.
• (01:23:40) The King of Fine-Tuning: Llama is no longer the default for enterprise fine-tuning. The Qwen 3 family has taken over because of the sheer variety of sizes (from 3B to 235B) and the performance of the 32B variant for cost/speed balance.
• (01:30:28) The Productivity "Hack": The most underrated workflow is dictating complex instructions into the ChatGPT mobile app. Speaking is faster than typing, and reading the output is faster than writing. It effectively acts as a high-fidelity thought translator.
• (01:33:48) Speech-to-Speech Leaders: As of late 2025, the best native speech-to-speech models are Gemini 2.5 Flash Native and OpenAI's GPT Realtime August Update.
• (01:45:00) The "Meters" Benchmark Trap: There is a bizarre marketing trend where companies brag about their agents taking longer (e.g., "Our agent worked for 12 hours!"). This misinterprets the "Meters" benchmark (which measures task complexity by human time). An agent taking 12 hours is like an employee bragging about taking two weeks to send an email.
• (01:47:47) The 100% Intelligence Cap: If a model hits 100% on the current Intelligence Index, it does not mean AGI or Singularity. It just means the current suite of tests (math, coding, science) is solved, and the benchmarks must evolve. We have already "solved" GPQA Diamond and competition math; the bar just keeps moving.

Now, let's dive into that all more in depth to fully explain everything we learned from this epic two hour talk.

Who Is Artificial Analysis?

Before we get into the meat of the conversation, let's talk about who these guys are and why their opinions matter (4:42).

Micah Hill Smith is a New Zealander. George Cameron is Australian. They met interning at Google "a long time ago." Both were building applications with AI models in 2023 when they noticed something was missing: good independent benchmarking of LLMs.

"In 2023, there was nothing that we could find where anyone was doing good independent benchmarking of LLMs," Micah explained. "And certainly not doing anything on speed and cost alongside different ways of thinking about intelligence."

The problem was obvious to both of them. When you're building applications with AI models, you're dealing with enormous ranges in cost and so many different options at each part of the stack. Without good independent benchmarking, making decisions was nearly impossible.

So they built Artificial Analysis as a side project.

"We probably had some naivety about how easy it would be to keep up to date," Micah admitted. "We were like, 'Yeah, we're going to spend like a month or two. We'll just build this thing.'"

They thought they'd update it maybe once a month as new models came out. Back then, a new model coming out every month was a big deal.

Now it feels like it's every couple of days.

What started as a side project became their full-time focus. Micah had actually quit his job at McKinsey to build a legal AI startup. He had law firms beta testing his product. George was working on other projects too. But developers loved Artificial Analysis so much that they decided to put everything else on pause and do it for real.

Today, they have around 20 people on their team and work with enterprises who subscribe to their analysis for navigating the AI landscape and adopting AI at scale.

The Wild Week That Changed Everything

We caught Micah and George at an interesting time. The eight days leading up to our conversation had been absolutely wild for AI releases (1:38).

Here's what dropped:

• Gemini 3 Pro from Google—the big news of the week before
• GPT 5.1 Max from OpenAI—trying to take the coding crown (and since then, GPT 5.2!) 
• Claude Opus 4.5 from Anthropic—"Anthropic Strikes Back"
• Plus a ton of smaller models, including Microsoft's computer use model

"Usually a ton of little models in between," Corey noted. "There's always, always, always the small stuff dropping here lately."

This is exactly why independent benchmarks matter more than ever. Every company releases their own benchmarks saying they're the "world's best coding model" or "world's best model." How do you actually know if they're telling the truth?

Google Finally Has the Best Model

Let's start with the headline: Gemini 3 Pro is currently the #1 model on Artificial Analysis's Intelligence Index.

Yes, Google. The company that seemed hopelessly behind for most of the AI race.

Micah pulled up their "Frontier Model Intelligence Over Time" chart, which plots each lab's leading model over time. The last two weeks looked like a colorful mess of leapfrogging (12:18).

"I love this crazy bit here," Micah said, pointing to the chart. "All those different colors. That's the last two weeks."

But here's what's remarkable: This was the first time Google has ever clearly had the best model according to Artificial Analysis's Intelligence Index.

"Before Gemini 2.5 Pro, I think it was fair to say that Google had never had a model that was competitive with the best at the time from OpenAI and Anthropic," Micah explained. "And by a pretty wide margin for a good while there."

Think about that. Google—the company where so many of the foundational AI innovations came from (transformers, anyone?)—spent nearly two years playing catch-up. It's "pretty nuts," as Micah put it, given that Google should have been in a very strong position going into all of this.

The Gemini 3 Pro preview release was "probably the largest jump in the intelligence index that we've seen in a while."

Claude Opus 4.5 came in second, right on Gemini's heels.

Credit to Google: Starting around early 2024, particularly with the Flash releases, they've been releasing models at a much more frequent cadence. The dots on Artificial Analysis's timeline are getting closer together.

Is AI Progress Stalling?

This brought us to one of the hottest debates in AI right now: Is there a "scaling wall"? Is progress slowing down?

Micah had two hot takes (19:00).

• ‍Hot Take #1: "Suggestions that progress has stalled have been greatly exaggerated."
• Hot Take #2: He sees "absolutely no reason to believe" that the clear upward direction isn't going to continue.

But wait—if you look at the chart, there's definitely a clustering of top models. Doesn't that suggest we're hitting some kind of ceiling?

Not necessarily. Micah argues that the clustering is better explained by the industry being more competitive than ever.

"Every corner for the last two and a half years, the industry has got more competitive, not less competitive," he said. "You can argue that there are a couple of players who have bowed out, but not in a material way at all. And we have seen the continual addition of new players."

When they first made their frontier model chart, they thought they could get away with four or five lines. Now they have 13 labs by default on the chart.

More players competing harder with enormous resources all trying to get to the top of the chart—that explains why everyone's bunched together at the top.

Corey made an excellent observation: "I think that comes through all the time. Even the entire GPT-5 discourse boils down to that—it being so soon after o3."

When you compare progress over two-week intervals, it feels incremental. But zoom out to a year? The progress is massive.

Just over a year ago, coding agents didn't really work. If you tried to get any LLM—especially before Claude 3.5 Sonnet—to do multi-step function calling, editing multiple files as a coding agent, it just fundamentally did not work.

That entire capability emerged in just over a year.

"It's so easy to compare progress from two weeks ago and be like, 'Wow, did anything happen these two weeks?'" Micah said. "But looking back to a year ago, we have come a huge way."

The Competitive Landscape

Speaking of competition, let's talk about who's in the race (20:06).

The frontier charts currently includes about 13 labs, including several Chinese AI companies that weren't on anyone's radar a year ago:

• DeepSeek - went from zero to being tracked as a serious player (released V3.2 since this recording).‍
• xAI - didn't even have a model when they did their 2024 survey.
• Moonshot (Kimi K2)
• Minimax
• Alibaba (Qwen)
• ByteDance

Companies to watch:

• Microsoft - just built a new data center with the largest single concentration of Nvidia GB300s.
• Meta - "They're the two I'm side-eyeing the most right now," Corey said.
• Amazon - investing heavily (just released a new round of Nova models at Re:Invent).

Re: Meta, the Llama 4 launch from Meta was "obviously less successful than they hoped," but as Micah pointed out: "They haven't released anything since early 2025 with Llama 4 Maverick. There are a lot of smart people and a lot of GPUs at Meta."

Notably absent from the frontier: Apple.

"If you'd have told me in November 2022 that inside of three years Apple wouldn't be on a list like this, I would have laughed out loud," Corey said.

Token Efficiency: Why Anthropic Models Feel Different

Now we get into the stuff that actually affects your wallet and your workflow (28:57).

Token efficiency refers to how many tokens different models use to complete the same task. And the range is enormous.

When reasoning models came along last year, there was a clear distinction: reasoning models used about 10x more output tokens than non-reasoning models. Nice and clean.

That distinction is getting blurry.

"Some of the highest token-using non-reasoning models use almost as many tokens—or more tokens sometimes—than some of the most token-efficient reasoning models," Micah explained.

Looking at their Intelligence Index runs across all 10 evals, token usage ranged from 5 million to 140 million tokens just among the highlighted models.

Here's where it gets interesting: Anthropic models consistently use far fewer tokens than competitors.

In reasoning mode, Claude models use around three times fewer tokens than models like Kimi K2 thinking or Grok 4.

"This is one of the reasons Anthropic models tend to vibe check well," George explained.

Think about it from three angles:

• Cost - You pay for tokens. Fewer tokens = lower cost.
• Latency - You wait for tokens. Fewer tokens = faster responses.
• User experience - Concise responses generally feel better than verbose ones.

When we tested Kimi K2 for creative writing (which everyone was raving about), we noticed something: it gives you the straight train of thought. No summary like ChatGPT or Gemini—just the raw thinking. "It's writing a novel in the thinking thread," as Grant put it (P.S: we just published a guide on how to setup Kimi to use as a creative writing partner via Fireworks here).

This is true of all open-weights reasoning models. If the weights are open, you can't hide the tokens. Developers can still build UIs that summarize the chain of thought, but the tokens are there.

The Thousand-X Cost Difference

Here's where things get wild.

The cost differences between AI models aren't 2x or 5x. They're 1,000x (36:25).

Artificial Analysis calculates the cost to run all 10 evals in their Intelligence Index once for each model. The results:

• GPT-5 Nano (minimal): ~$6
• Claude 4.1 Opus: Over $3,000

That's a 500x difference right there, and it doesn't even include the cheapest models.

Micah's car analogy: "You don't need to get a brand new Ferrari for the same thing you're getting a Prius for."

But as he quickly pointed out, the range in AI is way bigger than the car market. It's more like the difference between a Ferrari and a hundred Ferraris. Or as Corey suggested, "a Ferrari and a Lime scooter."

Why this matters for developers:

If you're building an app that needs to run 100-1,000 queries per day per user, you can't use Gemini 3 Pro or Claude Opus for everything. It would bankrupt you.

But here's the thing: for a bunch of use cases, you might get the same result from a model that costs 1,000x less.

The key insight for agentic workflows: You don't need to use the same model for every step.

"When you're building agents, it's not uncommon to have a series of different agentic stages within your workflow," Corey explained. "One task might be categorizing something—that's simple. For that, you grab GPT-5 Nano. But when you need to gather research and do something really important? Maybe you go to Gemini 3 Pro or Opus 4.5."

Use the expensive models where it counts. Use cheap models everywhere else.

AI Omniscience: The New Hallucination Benchmark

Artificial Analysis just launched something called AI Omniscience—their new knowledge and hallucination benchmark (44:53).

The concept is simple but important: Instead of just measuring what percentage of questions a model gets correct, they focus on reliability of knowledge.

Here's how it works:

• Models get +1 point for a correct answer
• Models get -1 point for an incorrect answer
• Models are told they should say "I don't know" if they don't know the answer

This creates a very different incentive structure than traditional benchmarks.

The questions are intentionally very hard—designed to be a frontier eval. Most models give incorrect answers more often than correct answers on this test.\

Why this design?

Traditional benchmarks reward guessing. If you have a 25% chance of getting a multiple-choice question right by guessing, and there's no penalty for being wrong, you should always guess.

But for factual questions in real-world use cases, the correct response when you don't know something should be "I don't know"—not a confident hallucination.

The benchmark lets them measure:

• Knowledge reliability (the main index).
• Percentage correct (accuracy score).
• Hallucination rate (how often models give wrong answers when they don't know).
• Breakdown by domain (42 subtopics across 6 main domains).

Hallucination Is an Incentive Problem

Here's Micah's key insight: "Hallucination is an incentive problem—and not just for the models, but for the labs and researchers.(49:31).

If all benchmarks reward "higher percentage correct is better," then obviously models should take a shot at everything. For many use cases, that makes sense. If you ask a model to write a poem, you don't want it thinking for a bit and saying, "Well, I'm not sure this is good enough, so I won't give it to you."

But for factual questions? If the model doesn't know the answer, the correct response should be to not hallucinate something.

This connects directly to OpenAI's recent hallucination paper (full paper). They essentially found that hallucination is built into the training process because models are incentivized to guess—just like students on a standardized test.

‍The hallucination rate results are fascinating:

Looking at their chart, Anthropic models have notably low hallucination rates. They're much less likely to attempt an answer when they don't know.

Interestingly, Llama 3.1 405B also performed well on this metric—a "massive dense model that had a very big pre-training run."

But here's what's concerning: Hallucination rates don't necessarily improve as models get smarter.

"All the Gemini models are grouped together," George pointed out. "Gemini 3 Pro is of course a lot smarter than 2.5 Flash and 2.5 Pro, but it has roughly the same hallucination rate."

Intelligence improvements aren't automatically translating to hallucination improvements. This suggests labs need to specifically optimize for not hallucinating—it won't just happen naturally as models get more capable.

Important clarification: These hallucination rates (80%+ for some models) don't mean models hallucinate 80% of the time in real-world use. The test is specifically designed with very hard questions to stress-test models at the extreme. But it's useful for comparing models against each other.

Which Models Know the Most About Programming?

This is where things get really practical for developers.

Artificial Analysis wrote questions based on the documentation and core libraries for various programming languages. It's not about writing code and seeing if it works—it's about whether models know all the stuff about the language (57:21).

"You want it to not have to Google 'what's the function called in one of the core libraries of Python,'" Micah explained.

The results show clear patterns:

For Python, Anthropic and Google models dominate. They've clearly paid enormous attention to making sure their models know Python cold.

"This is part of what drives them having the good vibes and getting things right the first time a lot of the time," Micah said.

For TypeScript (George's favorite), similar pattern.

But it varies by language. For Swift, the pattern is different—Google actually leads.

"If you're coding in Swift, use Gemini," was the takeaway.

This has broader implications beyond software engineering. George made the point: "This isn't just for software engineers. Companies out there are right when they say, 'Hey, this model—it's talked about as a general intelligent model, but it might not be great in law. It might not be great in civil engineering.'"

A lawyer working in civil law can look up how models rank specifically in civil law. The differences can be significant.

‍

The Value of Independent Benchmarking

We asked directly: Why does independent benchmarking matter?

The answer is simple but important (1:01:47).

When companies release models, they release them with their own internal benchmarks. Then independent third-party benchmarks come along and the numbers are almost invariably somewhat lower.

There are several reasons for this, and most of them aren't nefarious:

1. Cherry-picking benchmarks: If you're training a model and focusing on specific innovations, you're naturally going to highlight benchmarks that relate to what you were trying to do well. That's not hiding weaknesses—it's just marketing.

2. Optimization effects: You can often get a few percentage points better on most evals by putting effort into prompting and scaffolding. If you prompt and scaffold in ways similar to how you did some of your training, you get additional benefits.

3. Statistical noise: Maybe you didn't do enough repeats, or you had multiple checkpoints and picked the best one.

"None of this is dodgy or bad," Micah emphasized. "It's like Apple saying how long your MacBook's battery is going to last. Has your MacBook ever lasted that long?"

The point of independent benchmarking is that you run all the evals with identical methodology on all models (here's how Artificial Analysis does it). You pick tests that represent real use cases, upgrade them over time, and bring them together into something useful for comparison.

Image Generation: The New Leaderboard

Artificial Analysis doesn't just benchmark language models. They've been running an image arena for nearly two years, getting human preference data on text-to-image and image editing models (1:07:08).

The current leader? Google (Nano Banana Pro).

Specifically, the image output capability that came with Gemini 3 Pro (co-branded as "Nano Banana Pro"—a codename that stuck around from testing).

"That's now at the top of both our text-to-image and image editing leaderboards by a decent way," Micah said.

Looking at examples: things that used to be super hard for models—like getting text on a glass—are now possible with the latest couple of models. There's still room for improvement, but progress is clear.

The open-weights story mirrors language models:

The best open-weights image model has always been behind the best proprietary model, but has progressively caught up. The Flux models from Black Forest Labs have been particularly impressive, taking the baton from Stability AI.

"I love the Flux models," Corey said. "I still use them a lot."

Notably absent from the frontier recently: Midjourney.

"They haven't done anything frontier in a minute," we noted. Looking at the chart, it's been about a year since Midjourney had a competitive release. Some people still love their aesthetic, but the raw capability hasn't kept pace.

Video Generation: Where Things Get Interesting

If there's one modality to watch, it's image-to-video (1:13:45).

"Most of the creators pushing the limits of video generation models are almost exclusively focused on image-to-video," Micah explained.

Why? Because you can combine powerful image editing models to get exactly the starting frame you want, then use text instructions to create videos with much more precision.

The race has been incredibly competitive over the last year:

• Kuaishou's Kling models - Multi-image reference feature (upload multiple images to maintain character consistency across scenes).
• Google's Veo models - Native audio generation synchronized with video content.
• Pika - Special effects ("Pikaffects") that can inflate, melt, explode, or transform objects in videos.
• Minimax Hailuo - Strong performance in anime and cartoon-style video generation.
• ByteDance Seedance - Multi-shot narrative videos with seamless transitions between scenes (ranked #1 on Artificial Analysis benchmark).
• Vidu - Multi-entity consistency using up to 7 reference images for character/object placement.
• Luma Dream Machine - World's first HDR video model (Ray3) with reasoning capabilities for storytelling.

Many of these companies might not be familiar to Western audiences, but they're doing serious work.

What to expect over the next 6-12 months:

• Much more flexibility with multiple reference images
• Better controllability
• Innovation in sound generation (at the same time as video, or before/after)
• Ability to use separate text-to-speech models to create character speech
• Consistent characters, scenes, and locations across multiple clips

"We expect these to keep getting quite a bit better," Micah said, "and be enabling stuff that I'm going to be much keener to watch than 5-to-10 second clips taking up my social media feed."

Speech-to-Speech: The New Frontier

A year ago, speech-to-speech models were fairly described as "previews and beta tests."

Not anymore.

Current leaders:

• Google's Gemini 2.15 Flash Native.
• OpenAI's GPT Realtime (August update).

There's also a classic approach of combining Whisper (speech-to-text) with GPT-4o and OpenAI's TTS API to build voice agents.

New development: Alibaba dropped a Qwen 3 Omni model that does speech-to-speech. It's the first major open-weights speech-to-speech model.

"In our testing, it's not quite ready to probably use for actually building voice agents right now," Micah said. "But first major open-weights speech-to-speech model. That's pretty cool."

Not on the chart yet: xAI's Grok and Anthropic's voice capabilities.

Why? They don't have developer products in market yet for these. The consumer apps have voice features, but there's no API for developers to build with.

"If and when those companies decide to enter the market for developers and companies building stuff, then we would run Big Bench Audio," Micah said.

Pro tip from Micah: The speech-to-text in the ChatGPT app (both Mac and mobile) is very good. "I do a ton of dictating into ChatGPT on my phone as one of my ways of getting stuff done with AI."

The workflow that works best: Speak in, text out. It's faster to talk than type, faster to read than listen.

What Models Are Enterprises Actually Using?

Artificial Analysis surveys developers across a range of companies about what models they're using or considering using (here's an H1 2025 survey they did, for example).

Key finding: The average developer is using 4-5 different model families—not just one. (1:20:20)

2025 survey results:

• ~80% said OpenAI.
• ~80% said Gemini.
• Large jumps for Gemini (wasn't much reason to use it in 2024).
• DeepSeek went from zero to 53%.
• xAI went from zero (didn't exist) to significant adoption.

The models developers actually deploy:

Beyond the flagship APIs, a lot of companies are trying to deploy stuff themselves for various reasons:

• Don't want to send data to the public cloud.
• Want to fine-tune and do RL themselves.
• Want to minimize cost at scale.

For self-deployment, Qwen 3 models have become the go-to choice for fine-tuning—even more than Llama.

Why Qwen Dominates Fine-Tuning

Qwen 3 has essentially taken over from Llama 3.1 as the default choice for fine-tuning. Why?

Two reasons:

1. Pure performance. The models are genuinely good.

2. The variety. Qwen offers models at every size range:

• 700 million parameters
• 4B
• 7B
• 8B
• 30B
• 32B
• 235B (the massive MOE)

Different enterprises have different hardware constraints and budgets. With Qwen, you can pick exactly the size that fits your constraints.

"Often you're fine-tuning not for new capabilities, but because you want it faster or you want it cheaper," Micah explained.

The Qwen 3 32B model is a particularly common choice when you want something faster and cheaper than the frontier models.

"There are so many Qwen 3 models that it's actually one of the hardest families to go through on the site," Micah admitted.

Choosing a Provider: Speed vs. Cost

If you want to run an open-weights model like Kimi K2, you have options (1:40:08). Artificial Analysis tracks nine different API providers for Kimi alone.

They have a speed versus price chart that gives you an instant view of what each provider charges and how fast they serve tokens.

For Kimi K2, the standouts:

• Fireworks - fast, competitive price.
• Base 10 - fast, competitive price.
• Google Vertex - fast, competitive price.
• Parasail & GMI - both slightly slower, slightly cheaper.

Moonshot's own turbo endpoint is actually pretty expensive compared to third-party providers.

This analysis extends to proprietary models too. For Claude 4.5 Sonnet, you can compare:

• Anthropic's API
• Google Vertex
• Amazon Bedrock
• (Soon) Microsoft Azure

There are sometimes slightly different latency and speed profiles across providers, even for the same model.

What About World Models?

We asked about world models—the emerging capability to generate 3D environments you can move around in.

Micah's response: "I think we will be benchmarking stuff like that. I think those things are going to work pretty well."

He expects this to be a significant topic in about six months.

Current examples:

• Genie 3 from Google - impressive demos, though unclear how accessible it is.
• World Labs - you can try it right now.
• Skywork - has shown promising capabilities.

Some speculate that the reason Nano Banana (Google's image generation) is so good is because it uses similar technology to Genie 3.

What Models Do the Experts Use?

We asked Micah and George what models they actually use in their daily work.

George's setup:

• ‍For accessing cloud APIs: ChatGPT and Claude desktop apps are his go-tos. Generally faster than running locally.
• For local models and background tasks: "I really like GPT-OSS 20B because those tasks are usually agentic tasks. You usually want them for tool calling, not to respond to you like a chat application."
• GBT-OSS 20B is four-bit quantized, so the memory footprint is manageable, and it handles agentic work well.

Micah's setup:

• Uses "absolutely every model all the time" for work. Day-to-day go-tos are ChatGPT and Claude desktop apps on Mac.
• Has Claude hooked up to a bunch of MCP servers. Uses Cursor, Claude Code (here's a course from Anthropic to learn it btw), and a few other coding agents to try out different models.

A specific tip from Micah: The speech-to-text in the ChatGPT app is excellent. He dictates a lot into ChatGPT on his phone.

"Before AI, I didn't have that much use for speech-to-text because if you're sending a message to a person, you need to make it 100% correct. But when you're speaking to an AI agent and telling your coding agent to go do something, it doesn't matter. You can just dictate and the models are good enough that you don't have to fix anything."

Corey's local setup: Running Qwen 3 4B on his phone (iPhone 15 Pro, recently upgraded to 17). Uses LM Studio on desktop. Also a big fan of the speech-to-speech models—uses them in his Meta glasses and ChatGPT while driving.

The METR Task Doubling Benchmark

We also asked the guys about the METR task doubling benchmark that's gotten a lot of attention—measuring how long a task would take a human versus how fast AI can complete it (1:44:03).

Micah's take: "It's really cool. I quite like it."

But he was careful to contextualize it: "Like anything, it's one evaluation dataset. It would be misleading to say that because a model can do well on that, it can therefore do all tasks that would take a human an hour."

It's specifically for a particular subset of software engineering tasks.

The hilarious side effect: Labs have started bragging about how slow their agents are.

"At least four different companies in the last 3 months have said something about 'our agent worked for this long'—some crazy number like 'X model works for 12 hours on its own,'" Micah said.

This is absolutely not the point of the METR eval. The metric is how long did a human take and can an AI do tasks in that category. Bragging about your agent being slow is like an employee bragging they took two weeks to send an email.

"Come back in two minutes, then I'll be impressed," Micah quipped.

What Would a Perfect Score Look Like?

We asked a kind dumb question, but worth asking: What happens if a model hits 100 on the Intelligence Index? (1:47:47).

Micah's answer: "We promise not to let that happen."

Their job is to help developers and companies compare models and track AI progress. If models solved everything in the index, it would no longer be useful.

"There is some chance that next year models would achieve that if we didn't keep making any changes to the index," he admitted.

They've already seen some very hard evals get essentially solved:

• Competition math evals - models sitting at 99-100%.
• GPQA Diamond (very hard science eval) - "basically solved."

What would it look like in practice?

Models that hit 100% on the Intelligence Index would:

• Solve all the test-taking evals
• Solve the more use-case-focused coding/agentic/long-context reasoning evals
• Be "substantially smarter" and "more useful for a whole bunch of different things"
• Handle Terminal Bench Hard (very long-range tool use with terminal and computer)

"Models that would get 100% in Intelligence Index would be quite remarkable," Micah said.

But he was clear: "I would absolutely not say that they will magically be AGI or ASI or something and then some sci-fi takeoff or anything like that."

He doesn't put much stock in evals with dramatic names implying they'll be "the last exam." When the Intelligence Index reaches 100, there won't be some magic moment where everything changes.

"The lines are going to keep going up. It's going to be found. But it's not going to be some moment where the lines got to a certain point, there was a takeoff, and then everything changed."

Can AI Actually Do Our Jobs Yet?

This was the conversation closer, and it got real.

George flipped the question back: "Do you feel like these models are anywhere close to doing a substantial chunk of your work each week?"

Grant's honest assessment (1:53:17): No ('cause I've tried, y'all!)

For writing The Neuron newsletter:

• AI can't research the entire web and all the places he looks at without losing context or lacking access.
• It would have to compile that information and put it in place, hyperlinked in the right spot in the template.
• It couldn't then rewrite using his preferred style perfectly.
• And it couldn't put it into Beehiv (our newsletter sending platform) with all the formatting maintained

"I've tried to have AI automate my entire job, and it can do it—but not to the quality or the level that I want it to work with."

George's take on what's missing:

Compared to humans, models still struggle with:

1. Knowing when they don't know. Humans (at least when not lying or overconfident) are generally good at recognizing uncertainty.

2. Navigating ambiguity. When given a work task in a new environment, humans go out and ask questions to get information. Models don't do that—they just answer. They don't say "Hey, I need this information to get to an answer, otherwise I'm not confident."

"There's a bunch of these abilities that I think there's still a huge amount of progress to go on before we get to anything close that looks like AGI," George said.

The optimistic framing: "I think we've solved the one aspect of kind of raw compute intelligence."

12 months from now?

George asked if Grant thought he'd feel the same way a year from now.

Grant's answer: "Let's see if they make meaningful progress in user experience in the next 12 months."

The big gap is computer use becoming more commonplace. ChatGPT Agent is helpful but not mainstream (the dream product? A ChatGPT Agent / Claude Code hybrid that lives local on device and can control everything on your computer with your voice). Definitely by two years, he expects it to be really good.

Grant's specific wish: "Someone just needs to give me a UI where you can copy from one platform to another and maintain the formatting... Give the AI my words, have it edit it and maintain all the hyperlinks and all that stuff, then I'm like, okay, now I can just let AI write the newsletter and I can go drink a mimosa."

Micah's perspective:

"I'm pretty sure that the version of The Neuron that I want to be reading a year from now will not be completely written by AI."

He could give equivalent examples from all of their work about why it's actually so hard in practice to get these models to do useful work.

"These models do so much dumb stuff. Trying to get them to use a terminal, use a computer to do basic tasks can still be so painful."

But the capability gaps are visible. "I don't find it hard to imagine what a bunch of the missing capabilities could look like. How exactly they'll get filled in and what it'll look like is trickier."

Key Takeaways

Here's what you should remember from this conversation:

On AI Progress

• Progress hasn't stalled. The clustering at the top of benchmarks is explained by increased competition, not a scaling wall.
• Google finally has the best model. Gemini 3 Pro is #1 on the Intelligence Index, with Claude Opus 4.5 close behind.
• Zoom out. Two-week comparisons feel incremental. Year-over-year progress is massive.

On Model Selection

• Token efficiency matters. Anthropic models use ~3x fewer tokens than competitors in reasoning mode.
• Cost differences are 1,000x. Use expensive models where it counts, cheap models everywhere else.
• Different models for different tasks. Agent workflows should use different models for different stages.

On Hallucinations

• Hallucination is an incentive problem. Models are trained to guess, not to say "I don't know."
• Hallucination rates don't automatically improve with intelligence. Labs need to specifically optimize for this.
• Anthropic models have notably low hallucination rates. They're less likely to attempt answers when uncertain.

On Practical Usage

• Qwen 3 dominates fine-tuning. The variety of sizes makes it the go-to for self-deployment.
• Provider choice matters. Same model can have different speed/cost profiles across providers.
• Speech-to-text is underrated. Speak in, text out is the optimal workflow.

On What's Next

• Image-to-video is the modality to watch. Expect major improvements in controllability and consistency.
• World models are coming. Expect significant developments in ~6 months.
• AI won't fully replace creative work soon. But it will become much more capable as a collaborative tool.

About Artificial Analysis

Website: https://artificialanalysis.ai/

Founders: Micah Hill Smith and George Cameron

Team size: ~20 people

Business model: Free benchmarking data for developers; enterprise subscription for deeper analysis and reports

Started: 2023 as a side project; now a full-time company

This article is based on a live conversation recorded for The Neuron: AI Explained podcast. Watch the full episode on YouTube or listen on Spotify.