Microsoft's AI chief thinks your job has 18 months left. A new study says he's off by... a lot.
Last week, Mustafa Suleyman sat down with the Financial Times and dropped one of the boldest AI predictions we've heard in a while. The CEO of Microsoft AI said he expects "human-level performance on most, if not all, professional tasks" within the next 12 to 18 months.
Lawyers. Accountants. Marketers. Project managers. If you sit at a computer for work, Suleyman says AI is coming for your tasks by late 2027.
Suleyman defines his version of "AGI", or artificial general intelligence (where an AI can generalize to learn and do anything a human can) as "professional-grade AGI," specifically the ability for AI to handle tasks a regular professional does daily at a computer. In the interview, he also introduces his "modern Turing test" concept: could an AI take $100K, invent a product, market it, and turn it into $1M? He says models will satisfy this test "this year."
He's not alone. Anthropic CEO Dario Amodei warned last year that AI could wipe out half of entry-level white-collar jobs. Ford's CEO said AI would cut U.S. white-collar employment in half. The Atlantic called the coming disruption a "shark fin breaking the water." Last week, markets panicked with the "SaaSpocalypse," a software stock selloff triggered by Anthropic and OpenAI launching enterprise AI agents.
So... should you start updating your LinkedIn?
The reality check that just dropped
Enter the Remote Labor Index (RLI), possibly the most comprehensive test of whether AI can actually do real work. Published by researchers at the Center for AI Safety and Scale AI, it's a benchmark that doesn't test AI on trivia questions or coding puzzles. Instead, it tests AI on 240 real-world freelance projects, the majority sourced from Upwork.
These are projects that real clients paid real money for. Architecture plans. Game development. 3D product renders. Video editing. Data visualization. Audio production. Logo design. The median project took a human professional 11.5 hours and cost $200. Some topped $22,500 and 450 hours.
The researchers handed each project to six of the best AI agents available today (ChatGPT Agent, GPT-5, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, and Manus) and asked: can you deliver work that a real client would actually accept?
Also, the paper emphasized that previous benchmarks (like GDPval and HCAST) focused heavily on software engineering and writing tasks, where AI already performs relatively well. RLI is the first benchmark to test AI across the full diversity of remote work (23 categories including architecture, game dev, CAD, audio production, video). This is a big deal because it means prior benchmarks were measuring AI in its comfort zone.
IMPORTANT NOTE: The RLI study tested last-generation models, including GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro. None of today's frontier models (Opus 4.6, GPT-5.3, Gemini 3 Pro) were evaluated. That means the ceiling below is almost certainly higher now. How much higher? We don't know yet. But it's worth keeping in mind that this study is measuring where AI was, not where it is.
Here's what the paper actually tested:
Models used in RLI:
- GPT-5 (not 5.2 or 5.3)
- Claude Sonnet 4.5 (not Opus 4.5 or 4.6)
- Gemini 2.5 Pro (not Gemini 3)
- Grok 4
- Manus
- ChatGPT Agent
None of these are the current frontier. Every single model is at least one generation behind what's available now. And notably, they didn't even test the flagship-tier models from each company; they used Sonnet (Anthropic's mid-tier), not Opus. GPT-5, not 5.2/5.3. Gemini 2.5, not 3.
This doesn't invalidate the study, but it means the numbers below are a snapshot of last-gen capabilities, not current ones. The gap between these models and today's frontier (Opus 4.6, GPT-5.3, Gemini 3 Pro) is significant, especially for complex multi-step reasoning and tool use.
Even still, the results were... humbling
The best-performing agent, Manus, completed 2.5% of projects at a professional standard. Here's the full breakdown:
- Manus: 2.5% (earned $1,720 out of $143,991 in total project value)
- Grok 4: 2.1%
- Claude Sonnet 4.5: 2.1%
- GPT-5: 1.7%
- ChatGPT Agent: 1.3%
- Gemini 2.5 Pro: 0.8%
To put that in freelancer terms: if every project in the benchmark were a job posting, the best AI would earn $1,720 from $143,991 worth of available work. That's a 1.2% earnings rate. Not exactly "replacing white-collar work."
The paper also reports Elo scores showing meaningful differentiation between models, even though all scored well below the human baseline of 1,000 (Manus: 510, Grok 4: 468, ChatGPT Agent: 454, Sonnet 4.5: 442, GPT-5: 437, Gemini 2.5 Pro: 412). This matters because it shows measurable improvement model-over-model, even when the pass rate is still near zero.
Why AI keeps failing
The researchers analyzed roughly 400 evaluations to find out where things go wrong. The failures cluster into four buckets:
- Poor quality (45.6%): Even when AI finished the project, the work wasn't good enough. Think child-like drawings when a client wanted professional illustrations, or robotic voiceovers in a video that needed natural narration.
- Incomplete deliverables (35.7%): AI agents frequently submitted half-finished work. An 8-second video instead of 8 minutes. Missing components. Absent source files.
- Corrupted or unusable files (17.6%): Some agents produced files that simply couldn't be opened or rendered. Basic technical failures.
- Inconsistencies (14.8%): Especially with AI generation tools, outputs were visually inconsistent across files. A house that looked completely different in each 3D view. Design elements that clashed between deliverable files.
These categories overlap, meaning a single project could fail in multiple ways. The overall picture: AI agents don't just need to get smarter. They need to get more careful, more consistent, and better at verifying their own work.
The paper also mentions specific vivid examples: videos that were 8 seconds instead of 8 minutes, "child-like drawings using basic geometric shapes" when professional illustrations were requested, houses that looked completely different in each 3D view, and robotic voiceovers in videos that needed natural narration. Sounds not dislike hiring randos on Upwork without testing their skills in a paid assignment first. Pro tip: always do a trial assignment to test anyone you wanna hire on Upwork! Contractors get paid, and you get a free preview of their work. Win win.
Where AI actually succeeded
It wasn't all failures. AI agents matched or beat human output on a small subset of projects:
- Audio editing: Creating sound effects, separating vocals from music, merging voiceovers with intro/outro tracks.
- Image generation: Ad creation, logo design.
- Report writing and data visualization: Interactive dashboards, code-based visualizations.
Notice a pattern? These are tasks where the output is a single file type, the quality bar is more objective, and the work mostly involves text or code processing. The moment you need spatial reasoning, physical consistency, multi-file coordination, or subjective creative judgment at a professional standard... AI falls apart.
Worth noting the paper's finding that AI's successes cluster around tasks where the output is a single file type AND quality evaluation is relatively objective. The moment projects require multi-file coordination, spatial reasoning, or subjective creative judgment at professional standard, AI collapses. This is a useful framework for assessing your resulown work.
The gap between task-level and project-level automation
This is the key insight buried in the data. Previous AI benchmarks (including OpenAI's own GDPval study) showed AI approaching human parity on individual tasks like writing, web search, and administrative work. And that's real.
But there's a massive difference between "AI can write a decent paragraph" and "AI can deliver a complete architecture project with floor plans, 3D models, material boards, and electrical layouts that a paying client would sign off on."
The RLI researchers put it bluntly: prior benchmarks primarily focus on software engineering and writing tasks, but real freelance labor is far more diverse. RLI's projects averaged 28.9 hours of human work. The average completion time on previous benchmarks? Less than half that. Real jobs are harder, messier, and more complex than benchmarks suggest.
So who's right?
Suleyman isn't completely wrong. AI is getting better, and fast. The RLI's Elo rankings show consistent improvement from model to model. The trajectory is clear: AI agents will solve more and more of these projects over time.
But 18 months to automate "most, if not all professional tasks"? The data says that's wildly optimistic.
Consider the supporting evidence:
- A PwC report found 55% of chief executives saw no benefits from AI deployment.
- An MIT study found 95% of enterprise generative AI had no measurable impact on profit and loss.
- Another MIT simulation estimated AI could replace 11.7% of U.S. workers, not the majority.
- Even Suleyman admits the impact is mostly visible in software engineering so far, one domain out of dozens.
The honest answer? AI is a power tool, not a replacement worker. It's transforming specific tasks within jobs (drafting text, generating code, editing audio) while completely failing at the full scope of what those jobs actually require.
Suleyman also acknowledged in the interview that even in software engineering (AI's strongest domain), the role has "shifted to this meta function of debugging, scrutinizing, doing the strategic stuff like architecting." He's not claiming AI replaces engineers; he's saying the nature of the work changed. So even Suleyman's own framing is more nuanced than the headline prediction.
And even when the capabilities are there for the agent to completely automate complete tasks (like with OpenClaw or the Anthropic Agent SDK and today's top models), it requires technical know-how and security trade-offs to set up (meaning it's not exactly possible for big enterprises to lock down just yet).
Now, that might be one reason why OpenAI launched Frontier, a new platform to help businesses scale agents across their business. But unless you're a Fortune 500 or have a serious enterprise budget to throw around, this one probably isn't for you. So you'll still have to bootstrap your own agentic workflows or code your own agent swarms to get the true benefits of what today's AI can do.
What this means for you
There's the reality on the ground today, and then there's the reality going forward. We'll address both.
- Stop thinking about "will AI take my job?" That framing treats your job as one thing. It's not. It's dozens of tasks, some of which AI already does well (the 2.5%) and many it can't touch (the 97.5%).
- Figure out your 2.5%. Look at your actual workweek. Where are the repetitive, single-format tasks? First drafts, data summaries, code scaffolding, image generation, audio editing. Those are the tasks where AI genuinely saves you time right now. Use it there.
- Get very good at the other 97.5%. Ambiguity, judgment, quality control, client communication, multi-step creative projects, knowing when something "feels" right... these are the skills that every AI agent in this study failed at. They're also the skills that make you irreplaceable.
- Watch the trajectory, not the headline. Suleyman's 18-month timeline is possibly just hype. But the direction is real. AI agents are improving. The gap between 2.5% and 10% will close faster than the gap between 10% and 50%. The time to figure out how AI fits into your workflow is now, not when your boss hands you a mandate... or a pink slip.
- The real metric to watch? "Autoflation." The researchers also track "autoflation," which measures how much cheaper it gets to complete the same bundle of work using AI instead of humans. Right now? Under 4%. Meaning AI has barely dented the actual cost of getting real projects done. But like the automation rate itself, this number only goes one direction.
To end on a fun note: PC Gamer's Harvey Randall noted, Suleyman curiously omitted "Microsoft AI CEO" from his list of replaceable white-collar jobs. Funny how that works.
Read the full Remote Labor Index paper here. Watch Suleyman's full FT interview here.
