Claude Opus 5.5 Broke Our GPT-6 Astra Benchmark

Claude Opus 5.5 broke our benchmark.

That sounds like a dramatic way to describe a model test. In this case, it is literally what happened.

We went into our latest Neuron LIVE planning to run GPT-6 Sol and Claude Opus 5.5 through the same gauntlet. Build a black hole simulator. Make a Blender scene. Fix broken software. Adapt when the requirements change. Make increasingly absurd games. Same core instruction every time:

Don't explain how I could build this. Build it. Use the tools available to you, inspect your own result, and keep working until you believe it is finished.

Then we started using Opus 5.5 and the comparison got weird.

OpenAI's GPT-6 Sol is explicitly designed to bring more of GPT-6 Astra's capability into a cheaper, faster model for work at scale. Claude Opus 5.5 is Anthropic's frontier Opus release, aimed at the hardest coding, agent, and professional work Anthropic can throw at it.

Those are different optimization targets. And after hours of testing, the more interesting question stopped being "which model wins?" and became:

What happens when an AI agent gets good enough to keep creating, judging, fixing, and iterating long after the first prompt?

Short version: you stop benchmarking answers and start benchmarking taste, stamina, and whether the thing can actually finish a job.

First up, the TL;DR
The black hole was the cleanest test
Then we gave it Blender
CatDoom exposed the real question: who decides what to do next?
The 24-hour test made the human the bottleneck
So where does GPT-6 Sol actually fit?
Cost per token is starting to become the wrong scoreboard
The bigger shift is from generation to judgment
Watch the full stream

First up, the TL;DR

The stream started as GPT-6 Sol vs. Claude Opus 5.5. It turned into an Opus 5.5 demo reel because the models increasingly felt designed for different jobs.

GPT-6 Sol's pitch is efficiency. OpenAI launched it at $2 per million input tokens and $10 per million output tokens, while GPT-6 Luna drops all the way to $0.10 / $0.50. OpenAI says Sol and Luna bring much of Astra's capability into cheaper models that can support high-volume work.

Opus 5.5 costs more, but Anthropic is pushing a different edge: long-running, complicated work where the model has to decide what to do next instead of waiting for another human instruction.

Our tests made that difference visible:

It built an interactive black hole lab with gravitational lensing, time dilation controls, a photon lab, and explanatory material from one prompt.
It used Blender to create The Last Observatory, a miniature planet, observatory, astronaut, black hole, materials, lighting, camera, and animation.
It kept iterating on CatDoom, eventually producing a 3D Doom-style game with cats, weapons, levels, animations, and enough weird detail that we started asking copyright questions instead of "does it work?"
On a roughly day-long Dark Souls-style experiment, it kept building long enough that the bottleneck started becoming human judgment, compute, and whether we had clearly defined what "good" meant.

The most useful lesson was not "Opus good." It was that the benchmark itself has to change.

Can the model build? Can it inspect? Can it notice what is bad? Can it preserve the good parts while changing the bad parts? Can it work for hours without the human becoming the project manager every five minutes?

That is where agent evaluation is heading.

The black hole was the cleanest test

If you only try one demo from this stream, start with Anatomy of a Black Hole.

We asked Opus 5.5 to build a browser-based miniature science museum exhibit around a spinning black hole. It needed gravitational lensing, an accretion disk, camera controls, adjustable physics parameters, explanations, and one surprising interactive feature.

Then came the important part: run it, inspect it, fix what is broken, and do not stop at giving us code.

The result is an interactive ray-traced exhibit where you can change black-hole properties, watch light bend around it, fire photons through the scene, turn on a lensing grid, and even play with a "drop a clock" experiment.

That last sentence is doing a lot of work, so here's the mental model.

A normal chatbot task is mostly:

prompt -> answer

This test was closer to:

prompt -> plan -> write code -> run it -> look at the output -> notice problems -> change code -> run again -> decide it is finished

That extra loop is the agent part. And it is where the interesting behavior starts.

During the stream, Corey immediately found things we had not explicitly specified. The Photon Lab lets you aim light beams near the black hole. A lensing grid helps visualize how spacetime bends the apparent background. The interface explains that each pixel is effectively tracing a light ray through curved spacetime.

I asked for a science exhibit and the model apparently decided it also wanted office hours.

It was not perfect. Corey thought the teaching language got too technical in places. Some controls could have explained the underlying physics better. But that criticism is revealing because we were no longer asking "did the code run?"

We were asking whether the product taught the idea well.

That is a much higher bar.

Watch the black-hole section of the stream.

Then we gave it Blender

The next challenge was The Last Observatory.

This one asked Claude to use Blender to create an entire scene: a tiny planet, a miniature observatory, an astronaut, a black hole, materials, lighting, a camera setup, and a short reveal animation.

If you have never used Blender, the important thing to understand is that this is not one "generate image" button.

A Blender scene is a pile of coordinated decisions. Geometry has to exist. Materials need to be assigned. Lights need to be placed. Cameras need framing. Objects need scale and position. An animation needs keyframes. Then the whole thing has to render without looking like a fever dream assembled by IKEA instructions translated through four languages.

Opus handled enough of that stack that Corey immediately jumped to the obvious implication: one person can start doing animation work that previously required much more manual setup.

The finished scene still had obvious weak spots. The astronaut was less convincing than the planet and observatory. We wanted more reaction from the character. We were not even sure every lighting choice made physical sense.

But again, notice where the criticism moved.

We were art-directing.

We were no longer telling the agent which Blender button to press. We were judging composition, character quality, lighting, movement, and whether the scene conveyed the intended mood.

Watch The Last Observatory reveal.

CatDoom exposed the real question: who decides what to do next?

We have been making AI models build Doom with cats for long enough that this has accidentally become a benchmark.

Science is a process.

The current CatDoom site has three versions: the original 2D CatDoom, a full Doom-style episode, and the newer 3D CatDoom: Purrgatory.

What made the Opus run interesting was not that it could make an FPS. Models have been able to produce rough games for a while.

It was how much connective tissue Opus started inventing on its own.

The model created difficulty names. It wrote a premise about possessed cats. It added weapons, enemies, visual details, animations, environmental work, and kept iterating on things as tiny as paws, claws, and toe beans.

At one point, Grant's most interesting observation was that Opus seemed to be choosing which problems deserved attention.

That is different from code generation.

If the user says "make this better," there are thousands of possible next actions. Improve textures? Fix collision? Redesign a level? Add animation? Rewrite the UI? Optimize performance? Add sound? Stop?

An agent that can only generate waits for you to pick.

An agent that can operate starts making those decisions itself.

That is where Opus 5.5 felt different in our testing. It seemed more willing to keep a mental list of what still looked weak, pick a target, work on it, inspect the result, and move on.

Which is wonderful right up until you realize you have created a junior game studio that does not sleep.

Watch the CatDoom section.

The 24-hour test made the human the bottleneck

The most ambitious experiment came from an Opus-built Dark Souls / Elden Ring-style project inspired by Alex from Forward Future.

Grant used what Matt Shumer calls a "gauntlet loop." The agent breaks the project into parts, builds one, then a fresh-context critic reviews what exists. The system identifies the biggest gap, improves it, and repeats.

Builder -> critic -> gap -> fix -> inspect -> repeat.

That matters because long-running agents have a nasty failure mode: after hours in one context, they can become overly attached to their own previous decisions.

A fresh critic gives the system another set of eyes.

The run lasted roughly a day, although not 24 hours of completely uninterrupted autonomy. Grant still checked progress, handled permissions, and nudged it when necessary.

By the time we played the result, the interesting part was not whether it looked exactly like Elden Ring. It did not.

The interesting part was that the system had accumulated enough mechanics, environmental design, combat, movement, and unscripted ideas that it felt recognizably game-like.

Grant noticed features he had never explicitly requested and started wondering whether the model had invented them or pulled from available Unreal assets.

That uncertainty is important.

A flashy demo can hide a lot of scaffolding. Unreal Engine ships with assets and systems. A model using a mature game engine is not creating every pixel, animation, or mechanic from mathematical first principles. The useful question is whether it can orchestrate the available pieces into something coherent.

And that is exactly what human developers do too.

The difference is that this orchestration is becoming promptable.

Watch the Thornhollow section.

So where does GPT-6 Sol actually fit?

This is where the original "versus" framing falls apart.

OpenAI is positioning GPT-6 Sol and Luna around capability per dollar. Its September 22 release cut prices dramatically relative to the GPT-5.6 generation, and Sol is meant to bring serious reasoning and coding capability to work that needs to happen repeatedly and economically.

That matters a lot once you stop making demo projects and start running agents inside a company.

A software team does not need the most expensive model to classify every bug report, summarize every log, rename every variable, or update every simple webpage.

If you run 10,000 agent tasks a day, efficiency becomes capability.

We already made this argument in our GPT-6 Sol vs. Luna vs. Claude Opus 5.5 guide: there is no universal "smartest model" decision anymore. There is a routing decision.

A practical agent stack may look like:

goal -> cheap model handles routine work -> stronger model handles ambiguity -> frontier model handles the hard creative / judgment bottleneck -> human reviews what actually matters

That architecture is also why our recent OpenClaw 2.0 conversation matters here. Once people start running dozens of agents, the problem is not finding one magical model. It is deciding which intelligence to spend on which step.

Cost per token is starting to become the wrong scoreboard

Model pricing is easy to compare because it gives you a clean number.

$2 in. $10 out. $0.10 in. $0.50 out. Whatever.

But an agent job is not one token call.

It might plan, read files, call tools, fail, retry, inspect screenshots, reread context, run tests, ask another model to critique the result, then loop again.

So the metric we kept coming back to on the stream was cost per successfully completed job.

Imagine Model A costs half as much per token but needs three restarts and two hours of human babysitting.

Model B costs twice as much per token but finishes the job correctly while you go make lunch.

Which one is cheaper?

Welcome to enterprise software, where the answer to every pricing question is apparently "it depends on the workflow."

This is also the strongest counter-narrative to the "Opus is in another league" feeling from our demos.

These were deliberately ambitious creative tests. They reward autonomy, persistence, tool use, visual judgment, and willingness to keep iterating. That plays directly into Opus 5.5's strengths.

A company running millions of simpler agent steps may care far more about throughput, latency, reliability, and cost.

The models can both be excellent while winning different jobs.

The bigger shift is from generation to judgment

For the last few years, AI demos were easy to understand because the model made an object.

A paragraph. An image. A video. Some code.

Now the interesting systems are processes.

They create something, inspect it, decide what is wrong, choose a next action, preserve some work, throw other work away, and continue.

That means our tests need to change too.

We used to ask:

Did it answer correctly?
Did the code run?
Does the image look good?
Did it pass the benchmark?

Agent tests increasingly need to ask:

Did it understand the actual goal?
Did it choose sensible intermediate steps?
Did it inspect its own output instead of assuming success?
Did it preserve good work when requirements changed?
Did it fix someone else's mess instead of rebuilding everything?
Did it know when to stop?
How much human intervention did completion require?
What did the completed job cost?

That is why the stream stopped feeling like GPT-6 Sol vs. Opus 5.5.

One model made the economics of serious AI work more interesting.

The other made us ask how much work we can hand over before "using AI" starts looking more like directing a tiny software company.

And that last part is still very unresolved.

Opus could keep working for hours because we gave it tools, compute, and permission. That does not mean every extra hour made the project proportionally better. Long autonomy can also mean long detours, compounding mistakes, unnecessary polish, and expensive obsession over the wrong detail.

The next useful benchmark is not "how long can it run?"

It is how long can it run while continuing to make good decisions?

That is the test I want to see next.