What Scientists Can—and Still Can’t—Learn From AI’s Inner Workings

Imagine hiring someone who aces every interview question, then discovering you have no idea how they do the job.

Maybe they understand the work. Maybe they memorized the answers. Maybe everything falls apart when something unexpected happens.

That’s a version of the problem facing AI developers. A model’s answers tell us something about its abilities, but much less about the machinery producing them.

A field called mechanistic interpretability aims to understand those internal processes. A new research survey, Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety, examines whether that understanding can help make AI safer.

Its central question is refreshingly direct: “Does understanding how models work actually make them safer?”

The answer is cautiously promising. Researchers have increasingly useful ways to inspect AI systems. Turning those insights into dependable safety measures remains unfinished work.

What is mechanistic interpretability?

Modern language models learn from enormous amounts of data. Developers design the architecture and training process, but they don’t manually write every rule the finished model uses.

The result is a peculiar engineering situation: we can inspect the mathematical operations inside a model without fully understanding how those operations produce a particular behavior.

Think of receiving a spreadsheet with billions of interconnected formulas. Having every cell available doesn’t mean you understand the business it runs.

Mechanistic interpretability is the effort to reverse-engineer those internal processes. Researchers look for patterns that represent information, trace how those patterns interact, and test whether changing them changes the model’s behavior.

The word test is doing a lot of work there. An explanation needs evidence that it describes what the model actually does.

Consider a simple example: a model correctly answers that Paris is the capital of France. A conventional evaluation records a correct response. An investigation of its internal workings asks how it connected France with Paris—and whether the same process works when the question is phrased differently.

Both approaches are useful. They answer different questions.

Why passing an AI safety test isn’t the whole story

The gap between a model’s outputs and its internal processes becomes more consequential when researchers investigate harmful behavior.

In Anthropic’s earlier Sleeper Agents experiments, researchers deliberately trained models to behave safely in one context and insert vulnerabilities into code in another. Some of that behavior survived additional safety training.

These were constructed examples. They don’t establish how often such behavior would arise naturally. They demonstrate how reassuring test results can coexist with a hidden failure condition.

A follow-up experiment found that simple detectors could identify internal signals associated with the models’ triggered behavior. Looking inside revealed useful information that an ordinary interaction might miss.

Detecting a signal, however, doesn’t automatically explain its cause—or provide a reliable fix.

The survey repeatedly returns to that distinction.

How researchers look inside an AI model

AI models don’t organize information into tidy, labeled compartments. Researchers can’t simply open the “honesty” folder.

A single artificial neuron can participate in representing several different things, while one concept can be distributed across many neurons. Researchers call the first phenomenon polysemanticity. In everyday language, the same internal component can do several jobs.

Tools called sparse autoencoders try to untangle this activity into more understandable patterns, known as features. “Sparse” means the tool tries to explain a given internal state using relatively few active features.

Researchers can then investigate circuits: proposed pathways through which internal components work together to produce a behavior.

These approaches have produced striking demonstrations. In 2024, Anthropic identified millions of features within Claude 3 Sonnet, including patterns associated with recognizable concepts. Artificially strengthening some features changed the model’s responses.

That is meaningful progress. But assigning a recognizable label to a pattern is only the beginning.

An indicator light might reliably switch on whenever a machine overheats. Discovering that light gives you a useful warning. It doesn’t mean the light causes the overheating, or that disabling it repairs the machine.

Likewise, extracting information from a model’s internal activity doesn’t prove the model uses that information to produce its answer. Researchers need experiments that alter the relevant components and check what happens.

Five questions every AI explanation should answer

The survey’s most practical contribution is a proposed framework for evaluating explanations of AI’s internal workings. It asks five questions:

1. Faithfulness: Do the components identified as important have the predicted effects when researchers change them?
2. Completeness: How much of the relevant behavior does the explanation account for?
3. Stability: Does the finding survive changes in wording, experimental settings, or random starting conditions?
4. Causal validity: Is the component necessary for the behavior, sufficient to produce it, and specific enough to avoid disrupting unrelated abilities?
5. Human usefulness: Can people understand the explanation and use it to investigate or fix a problem?

Together, these questions push the field toward a tougher standard of evidence.

A colorful diagram may help explain an AI system. To support a safety claim, researchers also need to establish what it captures, what it misses, and when it stops working.

The authors acknowledge that their framework is a starting point. Running all these checks can be expensive, success thresholds need justification, and a method that works well on factual questions may struggle with complex reasoning.

The risk of explaining the wrong model

There’s another wrinkle: some interpretability tools build a simpler, more understandable approximation of the original model.

That approximation might produce similar answers while arriving at them through different internal processes. Researchers could end up explaining the substitute more successfully than the system they intended to study.

The paper calls this the proxy fidelity problem. Put plainly: how closely does the readable version preserve what the original is actually doing?

For safety, the stakes are clear. Failing to find a dangerous mechanism in an approximation doesn’t establish that it is absent from the original.

Imagine inspecting a simplified map of a building for hidden passageways. A passageway missing from the map could still exist in the building. You need evidence that the map preserves the details relevant to your inspection.

What this means for AI developers and businesses

The survey examines potential uses for interpretability, including editing specific knowledge, steering behavior by changing internal activity, and monitoring for warning signals.

It also covers research into how mechanisms develop during training and how future models might be designed to make their workings easier to inspect.

That ambition connects with The Neuron’s conversation with Goodfire CEO Eric Ho: making models easier to understand could make them easier to debug and improve deliberately.

For businesses evaluating AI, the research suggests a useful question to put to vendors: What does your explanation let you reliably detect or fix?

A strong benchmark score is useful evidence. So is a convincing explanation of a failure. The harder question is whether the explanation supports an intervention that keeps working across realistic conditions.

That is a practical implication of the research, rather than a commercial outcome the paper measures.

The authors argue for combining internal analysis with behavioral testing and other safeguards, while evaluating the contribution of each. Understanding part of a model can be valuable without providing a complete safety guarantee.

For anyone using AI, an explanation becomes valuable when it helps predict a failure, catch it, or prevent it.

The next meaningful advance in opening AI’s black box will be showing that what researchers learn inside helps systems behave more reliably outside the lab.

Authors and institutions

The paper’s 27 authors are Fazl Barez, Tung-Yu Wu, Hunar Batra, Gabriele Dominici, Florent Draye, Yanai Elazar, Damiano Fornasiere, Antía García, Abir Harrasse, Riley Harris, Sara Hooker, Michael Lan, Belinda Z. Li, Aaron Maiwald, Narmeen Fatimah Oozeer, Naomi Saphra, Or David Shafran, Adi Simhi, Robert Trager, Shriyash Kaustubh Upadhyay, Ziyu Yao, Yu Zhao, Shay B. Cohen, Mor Geva, Yoshua Bengio, Stella Biderman, and Hassan Sajjad.

Their listed affiliations span the University of Oxford; Martian; Università della Svizzera italiana; Max Planck Institute for Intelligent Systems; Bar-Ilan University; LawZero; Adaption Labs; MIT; Anthropic; Boston University; Tel Aviv University; Technion; George Mason University; University of Edinburgh; Mila and Université de Montréal; EleutherAI; New York University Abu Dhabi; and Dalhousie University.

Related reading

• Watch: Opening AI’s black box — Corey Noles and Grant Harvey speak with Goodfire CEO Eric Ho about understanding, debugging, and deliberately improving neural networks.
• Anthropic found Claude’s hidden workspace — A closer look at research into internal computations that happen before a model produces its answer.