AGI Is the Wrong Scoreboard. This 7-Level Framework Explains AI Progress Better

The word "AGI" is being asked to do too much work.

If this is the first time you've read the phrase, AGI stands for "Artificial General Intelligence." I'd define it, but that whole lack of concrete definition thing is going to come up later, so we'll get to that.

That's because the running line in tech is that "everyone has their own definition of AGI." To me, that means AGI actually just doesn't have a definition.

Which means it probably isn't a very helpful term to keep throwing around.

Let me give you an example:

Sam Altman recently described three things OpenAI is excited about: AGI accelerating research, AGI accelerating companies, and personal AGI helping everyone achieve their goals. 

That is a striking sentence because it quietly turns one giant concept into three different products, three different promises, and three different ways of decoding and measuring AI progress.

Chubby made the obvious-but-important objection underneath Sam’s post: the industry keeps saying AGI is within reach while using the same word to mean different things. Sam literally just did that three times in one sentence.

To make this concrete, a lab may use AGI to mean a system that can match humans across a broad range of cognitive tasks.

A company may mean AGI as a system that can run a workflow.

A normal person may mean a tool that finally helps them do the thing they were trying to do anyway.

That difference matters because AI does not arrive evenly (hence the popular term "jagged intelligence", or intelligence that is jagged in nature; expert in some areas, futile and pathetic in others).

And because we all have these rotating, nebulous, personal definitions for AGI, AGI for a mathematician, or for a startup founder, or a doctor, a lawyer, a designer, or a student will all show up at different times.

Why is this? Why do the goal posts keep getting moved? Because each person's needs require the model to “generalize” across a different series of domains and challenges... effectively creating a different slice of reality.

Kol Tregaskes put it simply: personal AGI happens when AI becomes broadly useful for you. A researcher reaches it when AI accelerates research. A company reaches it when AI transforms their personal operations. You reach it when AI helps you achieve your goals.

This idea could also be used interchangeably with the concept of getting "one-shotted" by AI: everyone has a use-case that once seen, will fundamentally change their opinion, belief in, and worldview on AI. You'll be completely blown away and convinced this is the future. And believe me: EVERYONE has one; if this hasn't happened to you yet, it's because your personal "benchmark" just hasn't been reached yet.

So why is this? When I read Kol's statement, it clicked for me: the term artificial general intelligence is perhaps the wrong frame, because everyone has their own human special intelligence, or HSI.

That's your unique set of skills, knowledge, and training that you've spent your whole life accumulating through experience, practical effort, or implicit observations. So AGI to you really means “general enough to do your specific skillset.”

Kol says Google DeepMind’s Levels of AGI paper is useful here because it tries to separate two things people often mash together: performance depth and generality breadth.

Breaking that down: how good is the system, and how widely can that goodness travel?

So the rough definition is:

AGI is an AI system that reaches human-level or better performance across a broad range of cognitive and practical tasks, rather than excelling only in narrow domains.

But how do you measure human-level performance? DeepMind’s key move was splitting AGI into levels instead of treating it as binary:

• Level 0: No AI or very narrow task tools.
• Level 1: Emerging AGI: equal to or better than an unskilled human across a broad range of tasks. The paper classified systems like ChatGPT, Bard, Llama 2, and Gemini as examples of this level at the time. Show you how old it is; Bard and Llama are goners.
• Level 2: Competent AGI: at least the 50th percentile of skilled adults across a broad range of tasks.
• Level 3: Expert AGI: at least the 90th percentile of skilled adults.
• Level 4: Virtuoso AGI: at least the 99th percentile of skilled adults.
• Level 5: Superhuman AGI: better than 100% of humans.

But even that framework does not fully answer the practical question most people actually have about AI and where it's headed (and how worried we should be about those two things). It's not like there's public leaderboards of every human and their skill stats for us to truly benchmark against. There's rough proxies like job title and degrees and the eliteness of an institution or company you work for. But unless you're in college or working for Meta, you're not getting stat-ranked against your peers like its the Hunger Games every few months. The problem becomes verification.

So the question we should be trying to answer is less “Has AGI arrived?” and more “Where can I trust AI enough to let it matter?”

The 7-level verification framework

That is where Greg Kamradt’s 7-level verification framework comes in. If you don't know Greg, he's the President of the ARC Prize Foundation, which is billed as the "North Star to AGI." Greg posted a kind of reaction to the recent breakthroughs in Math and Code involving verification.

To put it simply, AI is good at Math and Coding because those things have a verifiable right answer (code works or it doesn't; math is proved or it's wrong).

He said the concept of things being "easy to verify" got him thinking about the spectrum of difficulty in trying to verify other things.

We'll get into the framework he landed on below, but his core point is that AI progress moves fastest where reality can grade the work quickly. Code moves fast because you can run tests. Math moves fast because answers can be checked. Terminal tasks, data tasks, cybersecurity tasks, and structured professional work all improve faster because the model gets some version of an answer key.

In a brilliant essay titled "After Automation", Dan Shipper wrestled with the same problem: that the more you automate, the more work there is to be done because expert verification becomes the bottleneck. And before you can actually verify the work, someone has to frame the work. Benchmarks measure models inside frozen, human-made frames. So as the model climbs the frame, the frame saturates, and humans have to redraw the edges.

Put all these ideas together, and you get this idea:

AI progress is becoming a race to make more of the world affordably frameable, testable, and ultimately verifiable.

You can look at the billion dollar industry of expert networks built up around building better training data and benchmarks as the ultimate expression of this reality. In order to create a better player, you need to build a better scoreboard.

So perhaps instead of measuring progress towards this mercurial, elusive, impossible-to-measure goal of "achieving AGI", it would be more useful to reframe the pursuit of automation as the goal to bring formal verification (and therefore, predictability) to every aspect of modern life. I'm not saying that would be a good or right thing to actually go out and do, but it's a good way to measure progress towards something like making a total category of task "automation complete."

Greg’s ladder starts with work where feedback is nearly instant and ends with systems where the feedback loop can take decades to play out.

The higher you go up the ladder, the harder it gets to tell whether a decision or series of actions were actually right.

• Level 1: Instant, objective verification. Math, code, formal proofs, chess tactics, and parsing.
• • These can often be checked in seconds or minutes. The loop is tight, so AI can hill-climb quickly.
• Level 2: Fast but incomplete verification. Software engineering, UI implementation, data analysis, and security bug finding.
• • These can be tested quickly, but the tests only cover part of reality. “It passes tests” can still miss maintainability, edge cases, security exposure, and whether the next engineer will hate you in six months.
• Level 3: Human-evaluable creative work. Copywriting, design, thumbnails, sales emails, and landing pages.
• • You can get feedback from humans or markets in hours or days, but the signal is noisy. Taste shifts, metrics get gamed, and different audiences can disagree about what “good” means.
• Level 4: Market-verifiable work. Startups, investing, product strategy, hiring, pricing, and distribution get feedback over weeks, months, or years.
• • Reality eventually responds, but confounders pile up. A launch can fail because the idea was wrong, the timing was bad, the channel was weak, or the team executed poorly.
• Level 5: Experimentally verifiable science. Materials, biology, chemistry, medicine, and robotics.
• • These all have real ground truth, which makes them more objective than taste-driven work. The problem is that experiments cost time and money. AI helps most when it reduces the search space, proposes better candidates, or uses simulation to shrink the number of real-world trials.
• Level 6: Institutionally verifiable systems. Education systems, legal systems, city planning, and corporate management.
• • These all can be measured, but the feedback cycle is long and the counterfactual is hard. If a school improves five years later, was it the AI-designed curriculum, better teachers, different students, policy changes, or the local economy?
• Level 7: Civilization-scale verification. Governance, monetary systems, cultural norms, and geopolitical strategy.
• • These may take decades or generations to judge. The results are morally loaded, noisy, and hard to isolate. You may never get a clean answer... only accumulated historical evidence.

This ladder attempts to explain why AI can feel superhuman and strangely fragile in the same week. A model can crush a coding benchmark and still give mediocre advice on hiring, strategy, product positioning, or policy because those domains have slower feedback and messier definitions of “right.”

To Greg, that does not make the higher levels impossible to verify. But it means the next frontier is less about raw cleverness and more about building better frames, better tests, and better feedback loops to define "right".

Changing the frame to move up the ladder.

Also, verification has a cost. Kareem Carr summarized that cost as time, money, personal risk, and effort paid by the producer, consumer, and wider society. Greg agreed, and added another problem: some domains have “verification” without consensus. A novel can be evaluated by readers, critics, sales, awards, and vibes, but there is no objective answer key.

These all require a different kind of progress to measure: better frames, better tests, better simulations, and better feedback loops that compress the time between “the AI made a recommendation” and “reality showed whether it worked.”

Some even argue that there are things we should never attempt to automate, even if we can; to forever leave artistic work to the work of the creator, for example.

Dan Shipper argues that the "default model output" actually creates "demand for what’s different." He says "demand for what’s different is demand for human experts" even as we approach AGI. I'd put that slightly differently. Demand for what's different is demand for human specialists. Demand for HSI.

Even still, there are society wide outcomes and objectives we would love to achieve. Maybe not universally, but we know there are goals that individual communities and nation states and the entire global community can decide they want to achieve and work towards. Being able to verify, or make predictable, the outcomes of our actions to achieve those goals would be incredibly valuable.

This could also be one reason why Sir Demis Hassabis (of Google DeepMind) just suggested that AI could unlock a new "science of simulation", where we simulate these incredibly hard areas to verify (like monetary policy, or differing government strategies) in order to test them as close to emulating reality as possible in order to play out scenarios and attempt to make predictable incredibly hard decisions to game out.

But even if we never achieve full automation for all of these levels (or even 100% of any one level), just being able to define them could help us measure AI progress in a more robust way than current benchmarks or the pursuit of an undefinable AGI.

Is verification just training data?

Quick side note: When people talk about AI progress, they usually talk about training data.

That makes sense. The first big bottleneck was exposure: get enough text, code, images, videos, transcripts, papers, and examples into the model, and it starts to learn the patterns of human work.

But training data and verification are different layers of the same machine.

Training data tells the model: “Here is what humans have done before.”

Verification tells the model: “Here is whether this attempt worked.”

That second layer matters more as models move from autocomplete into agents. A model can learn the shape of a sales email from a million examples. It can learn the shape of a proof from papers. It can learn the shape of software from GitHub. Etc.

But once it starts acting in the world, or attempting to scale up the ladder to more fuzzy areas to verify, the important question changes from “Does this look like past completions?” to “Did this actually succeed?”

This is why code, math, and games scale so well. They generate their own answer keys. Code compiles or fails. Tests pass or fail. A proof checker accepts or rejects. A chess move wins, draws, or loses. In those domains, verification can become training signal quickly and cheaply.

Human feedback is also training signal, but it is slower, noisier, and more expensive. That is the basic logic behind reinforcement learning from human feedback, or RLHF: humans judge outputs, and those judgments help models learn what people prefer. Process supervision does something similar for reasoning, asking humans or systems to judge intermediate steps rather than only final answers.

The catch is that human judgment is not the same as the cold hard judgment of reality. A human can say a strategy memo sounds smart, but the market may later disagree. A doctor can say a clinical plan looks reasonable, but then the trial may fail. A teacher can say a lesson plan seems good, but student outcomes may take months to measure. Teachers only have so many months.

So while verification can become training data, the quality of that training data depends on the verifier.

The cleaner the verifier, the faster models can improve. The noisier, slower, more expensive, or more socially contested the verifier, the harder it is to turn experience into reliable progress.

That is why “data is the bottleneck” and “verification is the bottleneck” are starting to converge. The next generation of useful training data may come from building systems that can tell models, with increasing confidence, whether their work actually succeeded in these higher level domains.

This is also important for if (or when) an AI system is able to surpass collective human intelligence in the form of artificial superintelligence. At some point, the AI will have to move past our ability to generate enough data for us in order to predict in domains and areas where there is no training data, and no expert humans with enough context able to contribute meaningfully.

We're a long way from that, but whatever system we build will need to be able to fly without us at the helm. Like a NASA spacecraft that has to be able to land on MARS in a dust storm without any visibility, to reach level 6 or 7 our AI will need to be able to fly blind. To quote Yoda from the most controversial Star Wars film: "We are what they grow beyond." The student eventually surpasses the teacher.

Where the frontier is now

Artificial Analysis’s Intelligence Index is one of the cleaner public ways to watch the model race because it blends multiple (10+) hard evaluations across agentic work, coding, general reasoning, and scientific reasoning into one "intelligence score" (as well as breaking out where every modern ranks on each benchmark accordingly). Its methodology weights four buckets equally: agents, coding, general reasoning, and scientific reasoning.

The current leaderboard shows a crowded frontier. GPT-5.5 xhigh leads with a score of 60, GPT-5.5 high follows at 59, and Claude Opus 4.7 Adaptive Reasoning Max Effort sits at 57. Recent Artificial Analysis updates put Qwen3.7 Max at 57, Gemini 3.5 Flash high at 55, and Grok 4.3 high at 53.

The exact ordering will move (it often does). The bigger signal is that frontier performance is clustering. The gap between leading labs looks smaller than the gap between product choices, which Artificial Analysis also measures: speed, cost, context length, tool use, modality, and reliability.

Google’s Gemini 3.5 Flash is a good example. Artificial Analysis scored it at 55 on the Intelligence Index, but the more interesting detail was speed: more than 280 output tokens per second, with one million tokens of context and multimodal input support.

That makes it less like a trophy model and more like a workhorse model you can actually deploy inside products.

xAI’s Grok 4.3 tells a similar story from the cost side. It scored 53, improved sharply on agentic work, and cost roughly 20% less than Grok 4.20 to run through the full Artificial Analysis Intelligence Index despite producing more output tokens.

Then there is the app layer. Cursor Composer 2.5 placed third on Artificial Analysis’s Coding Agent Index behind Claude Opus 4.7 in Claude Code and GPT-5.5 in Codex, while costing pennies per benchmark task in standard mode. Cursor says the base model is Kimi K2.5, but about 85% of the compute behind Composer comes from its own additional training and reinforcement learning.

That is a huge clue. The next jump may come from better systems around models, not only smarter base models.

The hidden variable is verification

The model race makes more sense if you stop asking one broad question, “Which model is smartest?” and start asking a sharper one: How cheaply can this system check whether its work is good?

Artificial Analysis’s methodology shows an interesting pattern. The index is text-only and English-language. It runs standardized, zero-shot, instruction-prompted tests. It also uses pass@1 scoring, which means the first answer counts.

That is useful. It is also a map of what can be measured. The easiest gains show up where the evaluator can say yes or no without waiting for the world to respond.

You can think of the verification ladder in four rough bands:

• Cheap verification: math, code, formal proofs, parsing, chess tactics, terminal tasks.
• Medium-cost verification: UI implementation, data analysis, bug finding, professional deliverables, and customer support.
• Expensive verification: markets, hiring, pricing, product strategy, scientific experiments, robotics, and medicine.
• Brutally expensive verification: education systems, legal systems, city planning, corporate management, governance, culture, and geopolitics.

Expensive here doesn't just mean raw cost, it also accounts for time spent, personal risk involved, and total effort required to complete.

Right now, AI is racing through the first band, pushing into the second, nibbling at the third, and mostly simulating (likely inaccurately) the fourth. 

There are other benchmarks as well:

Single-question benchmarks are useful, but agent progress depends on how long a model can keep working without losing the plot.

That is why METR’s time horizon work matters.  Time Horizon 1.1 work. METR defines a model’s 50% time horizon as the length of tasks, measured by how long they take human professionals, that the model can complete autonomously about half the time. In the 2026 update, METR expanded its task suite from 170 to 228 tasks, doubled the number of eight-hour-plus tasks from 14 to 31, and estimated a post-2023 50% time-horizon doubling time of about 130.8 days.

That is useful because it turns “agents are getting better” into something closer to “agents are handling longer work.” But METR’s own cross-domain analysis also shows why time is only one axis. Software, scientific QA, math contests, and competitive programming clustered around 50-200+ minute horizons with fast doubling times, while visual computer-use tasks were 40-100x shorter, and self-driving improved more slowly. METR also cautioned that benchmarks are easier than corresponding real-world tasks because they are cheap to run and score.

Greg’s own organization, ARC Prize, is pushing on a different missing axis: can a model discover the frame at all? ARC-AGI-3 tests agents in novel, abstract, turn-based environments where they have to explore, infer goals, build internal models of the rules, and plan actions without explicit instructions. The paper reports that humans solved 100% of the environments, while frontier AI systems scored below 1% as of March 2026.

That is a useful contrast with Artificial Analysis. Artificial Analysis measures performance inside standardized text prompts. METR measures how long a model can keep executing inside a task. ARC-AGI-3 asks whether the model can figure out the task’s hidden logic in the first place.

For personal risk and communal effort, no such benchmark exists. The phrase you can't improve what you cannot measure comes to mind. That absence is part of the point. The closer a task gets to medicine, law, hiring, governance, or city planning, the less “score the answer” works.

This kind of evaluation starts to look more like risk management. NIST’s AI Risk Management Framework, for example, frames AI risk as something to manage across individuals, organizations, and society, which is the right level of analysis for the higher rungs. So how do you turn that into a leaderboard?

The missing piece: who creates the frame?

Dan Shipper’s essay After Automation makes the verification argument better because it catches something easy to miss: benchmarks measure work after a human has already turned the problem into a frame.

A benchmark prompt decides the role, inputs, success criteria, output format, and hidden assumptions that human special intelligence typically contains. It freezes a messy real-world problem into a static, measurable shape that the model can then climb that shape.

Dan uses OpenAI's GDPVal eval as an example: a model completing an expert audit spreadsheet task looks like expert work. But the prompt already specifies the spreadsheet, confidence level, tolerable error rate, sampling criteria, risk areas, output tabs, and formatting for the AI, so really, the human expert did most of the heavy lifting here to frame the problem.

A lot of human special intelligence is already inside the setup before the model begins its work; the AI just executed on their behalf.

This does not make the progress fake. But it makes the progress more specific. The model is doing real work, but only after someone has created a target clean enough to evaluate.

So the real unit of AI progress is:

• Can the task be framed clearly?
• Can the output be checked cheaply?
• Can the system learn from that check?
• Can humans trust the result enough to let go?

This also explains why automation can create more expert work, which is Dan's point: making output trivial requires more expert work to differentiate from a sea of sameness.

When AI makes yesterday’s baseline level of intelligence (or call it task competence) cheap, more people try to do it. They try to code, design, analyze, support, and strategize. Output explodes.

Then the scarce work moves up a level: deciding what is worth doing, what fits the situation, what should be preserved, what should be deleted, and what “good” means now.

AI climbs the frame. Humans are the framers. So moving the goal posts is actually the most important part of it. You're moving up the ladder.

Or, to quote the AI industry's favorite term: you're hill-climbing. Descending the gradient? Rising through the charts? Pick your metaphor, any metaphor!

The most important metric may be task length

METR measures the human expert task duration at which an AI agent succeeds with a given reliability. A 50% time horizon does not mean the model literally works for that long. It means the model can solve tasks that would take a human expert roughly that much time about half the time.

METR’s original finding was that AI agents’ task-completion horizons had doubled about every seven months over roughly six years. Its updated Time Horizon 1.1 work suggests the post-2023 trend may be faster, with a 50% horizon doubling time of about 130.8 days. Post-2024 estimates are even faster, though METR is careful about including caveats.

METR’s current task suite is concentrated in software engineering, machine learning, and cybersecurity (low level stuff, according to Greg). The tasks are self-contained, well-specified, and automatically evaluated. That makes them a strong signal for agentic progress in verifiable domains, and a weaker signal for messy work where success depends on unclear goals, hidden context, persuasion, politics, or timing.

Still, the trend is the trend. If task horizons keep extending, the frontier moves from “answer this question” to “finish this project.” That is a much bigger change for knowledge work.

GDPval is the bridge into real work

OpenAI’s GDPval was designed to push evaluation closer to economically useful work. It includes 1,320 tasks across 44 occupations and nine major industries, created and vetted by professionals averaging more than 14 years of experience. The deliverables include documents, slides, diagrams, spreadsheets, blueprints, legal briefs, support conversations, and care plans.

That is closer to work than most academic benchmarks. GDPval-AA, Artificial Analysis’s version using an agent harness, currently has GPT-5.5 xhigh, Claude Opus 4.7 Adaptive Reasoning Max Effort, and GPT-5.5 high clustered near the top.

OpenAI’s own writeup says frontier models are approaching the quality of industry experts on some tasks, and can complete them roughly 100x faster and 100x cheaper in inference terms. But OpenAI also makes the limitation clear: GDPval is one-shot. Real work usually involves context, iteration, meetings, ambiguity, unstated preferences, and someone changing their mind halfway through.

That makes GDPval both exciting and bounded. It shows that models can generate plausible professional artifacts at high speed. It does not prove they can own messy business outcomes from start to finish.

Dan’s frame argument sharpens the point: GDPval measures what happens after professionals have already turned work into a well-scoped deliverable. The next leap is getting AI systems to help create, test, and revise those frames in live workflows.

Science is where the ladder gets interesting

Science sits in a strange spot on the verification ladder. It has ground truth. For example, physics does not care about how you feel. You follow the laws of physics, or you fail. But experiments are slow, expensive, and often hard to reproduce.

That is why the next major AI push is likely to focus on shrinking the experimental loop. Google DeepMind’s recent Gemini science case studies describe models helping researchers solve open problems, refute conjectures, generate proofs, and use code to verify complex derivations.

This is where simulation becomes strategically important. If you can simulate a molecule, a robot, a market, a classroom, or a city well enough, you can test more ideas before the cost of reality makes it prohibitively expensive.

But simulation is only as good as the assumptions inside it. A simulator can compress feedback when the world model is strong. It can also create a beautifully optimized fantasy league for bad ideas.

That is the core tension for levels five through seven of the verification ladder. AI progress in biology, robotics, medicine, education, management, and governance depends on turning slow feedback into faster feedback without losing contact with reality.

Do we need to move to realtime benchmarks?

I think where all of this is headed is some kind of real time benchmark, equivalent to a real time feed of agent performance that we can use to judge how AI is doing on an ongoing basis.

Eventually, all agentic systems are going to be running 24/7 in some capacity to be maximally useful, so keeping track of them will become even more important than it is today.

How will we move from the realm of static benchmarks to realtime benchmarks? It will require observability of not only the agent's activity, but the outcomes it produces.

This will require a level of transparency that may be uncomfortable for most. Perhaps it turns into a world where every company has its own private and public realtime benchmarks for their agents and that the outcomes those agents are graded against, both internally and by the public.

We already have early versions of this in government data. They are imperfect, delayed, and narrow, but they show the shape of the system.

For example, the Atlanta Fed’s GDPNow is a near-real-time economic benchmark. Official GDP arrives with a delay, so GDPNow estimates current-quarter GDP growth as new source data comes in. The Atlanta Fed says the model updates six or seven times a month and uses no subjective adjustments once it begins forecasting a quarter.

The New York Fed’s Weekly Economic Index goes one step closer to the idea. It's built from daily and weekly indicators and is designed to signal the state of the U.S. economy at a higher frequency than traditional GDP. It combines ten indicators covering consumer behavior, labor, and production.

Weekly jobless claims are another version of this. FRED’s initial claims series, sourced from the U.S. Employment and Training Administration, updates weekly and tracks new unemployment insurance claims. That makes it one of the closest things the government has to a recurring pulse check on labor-market stress.

You can find similar fragments elsewhere:

• USAspending’s API gives the public access to federal spending data, including awards, contracts, grants, geographic breakdowns, agency breakdowns, and account-level spending.
• The CDC’s wastewater dataset gives public-health officials a faster signal of disease spread than waiting for everyone to get tested, report symptoms, or show up at hospitals.
• IRS filing-season statistics track returns received, returns processed, refunds issued, refund amounts, and web usage during tax season. That is a crude but useful benchmark for service throughput inside a giant public system.
• Performance.gov is the U.S. government’s public performance site, though it is closer to a management dashboard than a real-time benchmark.

None of these are “government AGI benchmarks”, obviously. They are more like civic vital signs. But that's the point.

If AI agents are eventually running inside agencies, hospitals, classrooms, courts, infrastructure systems, and companies 24/7, the important question becomes: What live outcomes are they being graded against?

For companies, we already have rough public benchmarks. The stock market is a real-time, vibes-heavy benchmark for expected future performance. Quarterly earnings are a harder check, but they arrive more slowly. Private benchmarks include customer churn, support resolution time, product usage, revenue retention, defect rates, and even social media backlash are better benchmarks because they are closer to actual operations.

Governments need their own version of that stack.

A real-time civic benchmark system would probably need at least four layers:

• Economic pulse: realtime GDP nowcasts, inflation nowcasts, jobless claims, wage growth, business formation, bankruptcy filings, and tax receipts.
• Service delivery: realtime wait times for gov services, application backlogs, error rates, appeal reversals, call-center resolution, benefit delivery speed, and citizen satisfaction.
• Public-risk signals: realtime hospital capacity, wastewater trends, crime incidents, infrastructure failures, fraud reports, disaster response times, and safety incidents.
• Long-outcome checks: realtime cohort-based learning gains, health outcomes, housing stability, mobility, trust in institutions, fiscal sustainability, and inequality.

The hard part is that higher-level benchmarks add risk. When a benchmark becomes a target, institutions start optimizing for the number instead of the outcome. A city can reduce “average response time” while ignoring hard cases. A school can raise test scores while narrowing what students learn. A benefits agency can lower fraud while wrongly blocking eligible people.

That is why personal risk and communal effort are harder to benchmark than time. METR can measure task length. GDPNow can nowcast GDP. But “how much risk did this decision impose on a vulnerable person?” and “how much coordination did this policy require across a community?” are harder to compress into one number.

The closest existing frame is risk management, not a leaderboard. NIST’s AI Risk Management Framework is explicitly about managing risks to individuals, organizations, and society, which is probably the right shape for high-stakes public systems.

On the social front, you can think of social media feeds and engagement and follower counts as a kind of realtime social benchmark. Obviously the scary version of this is a social credit system, but today's social media followings still come with inherent judgements even if they aren't explicit. Job titles, degrees, your circle of friends, your credit score, your bank account; these are all proxies for social benchmarks as well. One man even actually benchmarked his entire life for 10 years, and made it public.

So the future may look less like one numeric benchmark and more like many nested dashboards, akin to the Artificial Analysis Intelligence Index, but for everything:

• private benchmarks for agents inside companies,
• public benchmarks for agent systems affecting customers or citizens,
• regulatory benchmarks for agent safety, fairness, and failure modes,
• and civic benchmarks for whether institutions themselves are actually improving lives.

To make AI useful at higher levels of the ladder, we need observability of the agent’s actions and the outcomes those actions produce. That creates better feedback loops. It also creates surveillance risks, ranking games, and proxy traps.

The next frontier is building real-time benchmarks that make institutions more accountable without turning every human life into a dashboard.

I am definitely biased as a person who grew up playing video games, but why don't we benchmark more things in the first place? As humans, we love chasing achievements... we could stand to benchmark a bit more and track progress accordingly. Even if the benchmark is just maintaining a steady state, it helps to track things (just not everything, according to the dude who actually did track everything).

And if we're benchmarking agent capabilities anyway, why not benchmark them against the things we actually care about?

Where benchmarks can lie

The strongest counterargument is simple: benchmark progress can look cleaner than real-world progress.

Humanity’s Last Exam exists because many older academic benchmarks became too easy for frontier models. A separate benchmark saturation study found that nearly half of 60 analyzed language-model benchmarks had saturated, and that hiding test data does not reliably protect a benchmark from getting used up.

Cybersecurity shows the same gap from another angle. The UK AI Security Institute reportedly found that Anthropic’s Claude Mythos completed a previously unsolved cybersecurity test in three out of ten attempts, a milestone for autonomous cyber capability. At the same time, the Cyber Defense Benchmark found that frontier models performed poorly on open-ended threat hunting, with the best model finding only a small share of malicious events on average.

Both can be true. AI can improve rapidly on structured cyber ranges and still struggle when the problem becomes open-ended, evidence-driven, and operationally messy.

That is the main reason to be careful with every frontier chart. A benchmark measures the part of intelligence that a benchmark can see. The more a task depends on context, taste, institutions, markets, or delayed outcomes, the harder it becomes to know what the score means.

The hardest benchmark is human flourishing

The closest thing to a civilization-level benchmark that already exists, at least in early form, is human flourishing.

The Global Flourishing Study measures whether people are actually doing well across domains that GDP mostly misses: happiness, health, meaning, character, relationships, and financial stability. That makes it a better north star for governments than economic growth alone. A country can get richer while its people become lonelier, sicker, more anxious, or less connected.

The problem is frequency in reporting. A flourishing survey is usually slow, expensive, and self-reported. It can tell you whether a society is working, but not always fast enough to help you steer it in real time.

There are also early attempts to aim AI evaluation directly at this target.

The Flourishing AI Benchmark tries to measure whether AI systems support human flourishing across seven dimensions: character and virtue, close relationships, happiness and life satisfaction, meaning and purpose, mental and physical health, financial and material stability, and faith and spirituality. It uses 1,229 objective and subjective questions and tested 28 leading language models. The top model scored 72/100, and the authors found that no model was acceptably aligned across all dimensions.

That is closer to the kind of benchmark we actually need at the top of Greg’s ladder. It asks a different question than “Can the model complete the task?” It asks whether the model is helping a human life go better.

That is why the Human Flourishing Geographic Index is interesting. It uses LLMs to classify roughly 2.6B geolocated U.S. tweets into county- and state-level indicators of flourishing-related discourse.

It is imperfect, because social media is a distorted mirror of human life. But it points at the shape of the next benchmark: not one annual score, but a living map of whether people are becoming healthier, safer, more connected, more economically stable, and more capable of pursuing meaningful lives.

That is also where the verification problem gets morally dangerous.

When you measure code, the code does not care. When you measure citizens, the measurement changes the society being measured. A “real-time flourishing benchmark” could help governments detect suffering earlier, allocate resources better, and notice when policy is failing.

It could also become a surveillance machine, a social-credit proxy, or a dashboard that rewards institutions for making people legible instead of actually helping them flourish.

So the final rung of the ladder is not “Can we measure civilization in real time?”

It is: Can we build feedback loops for human flourishing without turning human beings into the raw material of the benchmark?

A newer paper, Positive Alignment: Artificial Intelligence for Human Flourishing, pushes the idea further. Seb Krier, Ruben Laukkonen, and their coauthors argue that AI alignment should move beyond harm prevention, compliance, and controllability toward systems that actively support human and ecological flourishing in a pluralistic, context-sensitive, user-authored way.

That phrase matters: user-authored. The scary version of flourishing benchmarks is a central authority deciding what “a good life” means and scoring everyone against it. The useful version is a set of feedback loops that help people, communities, and institutions define their own goals, then notice whether AI systems are actually helping them move toward those goals.

So maybe the final benchmark is not one leaderboard at all. It is a stack of living, contested, human-governed benchmarks: some personal, some organizational, some civic, and some ecological. The point would be to make AI accountable to the outcomes humans actually care about, without pretending there is one universal score for a good life. Sounds right to me!

What does all this do to the AGI debate

For most people, "AGI" will be personal. Just like everyone has their own AGI definition, everyone will have their own AGI moment. That's the moment where the model one-shots you and crosses the threshold into meaningfully impacting your day to day (be it your work or personal use cases). The model becomes good enough inside your frames, for your goals, with your verification loops. It is functionally "AGI"... whatever that means.

That is why two people can look at the same model and disagree completely on its efficacy. A musician may already feel like they have personal AGI for composition. A robotics researcher may feel nowhere close. A company may have AGI for support triage and still have no AGI for strategy. They are standing on different rungs of the ladder.

If we're able to measure things correctly, eventually everyone will have their own personal realtime benchmarks. Those benchmarks will be personal to them and their own goals, and flexible enough that they update as those goals change. And as we move up the ladder of verification, we can track not only whether or not an AI can do something correctly, but whether or not the AI doing that thing actually leads to positive outcomes; for us personally, and for civilization as a whole.

What to watch

As we track AI progress, we should ask ourselves: Is the world becoming more frameable, testable, and verifiable?

To answer, watch for six things:

• Longer agent horizons: Can models reliably finish multi-day projects, or do they still need humans to reset context every few hours?
• Better frames: Can systems help humans define the right goal, scope, inputs, constraints, and success criteria across each level of the ladder? Do we know what we want to achieve in the first place?
• Cheaper expert grading: Can expert rubrics, automated judges, and audits make professional work more measurable without turning it into checkbox theater? Are the outcomes verifiably correct and also unique?
• Better simulators: Can labs build simulations that predict real experiments, real users, real markets, or real institutions? And can we collect the data we need in order to do this properly?
• Closed-loop systems: Can AI propose, test, observe, revise, and try again without a human manually stitching the loop together?
• Deployment evidence: Do the models change revenue, health outcomes, bug rates, discovery timelines, student learning, or customer behavior? In order to judge success, we first must define what objectives to go after, and update them according to the changing world we live in.

Our take: the most important AI progress over the next few years may come from better frames and better answer keys. What if we just start benchmarking everything we do so we can define our objectives to align our incentives?

I find most long running issues I observe in society are inherently an incentives problem; show me the incentives, and I'll show you the outcomes. In order to change the outcomes, we must change the incentives.

The raw models will keep improving. Some will be faster. Some will be cheaper. Some will win specific leaderboards by a few points. But the systems that matter most will be the ones that can turn fuzzy work into useful frames and reliable feedback loops.

That leaves one very real open question: Can we build frames, simulators, and evaluators for the hardest parts of reality without optimizing for the wrong proxy?

If the answer is yes, AI progress moves from code and benchmarks into science, business, and institutions much faster (and more effectively) than most people expect. If the answer is no, we will get lots of systems that perform beautifully in synthetic worlds... and then discover that reality refuses to grade on a curve. A beautiful demo that falls apart in production.