NVIDIA’s Plan to Stop AI Agents From Wasting Expensive Memory

An AI agent can spend seconds writing code, then pause while a tool runs.

During that pause, its working context may still occupy some of the most expensive memory in the data center.

Multiply that across thousands of agents researching documents, testing software, and carrying information between steps, and you get a problem that buying faster chips alone won’t solve: too much valuable memory holding work that is temporarily doing nothing.

That’s the useful takeaway from NVIDIA’s new explanation of its approach to AI memory. The company’s argument is that GPUs, CPUs, networking, storage, and serving software need to be designed together so more memory goes toward active work.

For anyone trying to turn an impressive agent demo into an affordable production service, this is where the infrastructure story gets interesting.

What an AI model needs to keep around

Running a language model requires memory for two different things.

First are the model weights: the learned numerical values that encode the model’s capabilities.

Then there’s the KV cache, short for key-value cache. It holds intermediate attention calculations from previously processed tokens, allowing the model to reuse that work as it continues generating a response.

Think of weights as the model’s learned equipment and the cache as the working material spread across its desk.

That desk gets crowded quickly. An agent may read a document, call a tool, inspect the result, and then decide what to do next. Each step can add context that subsequent steps need.

This working state is also different from a product’s “memory” feature that remembers your preferred writing style across chats. Here, we’re talking about the machinery that makes an active inference session efficient.

NVIDIA’s video explainer walks through the five techniques it uses to manage that machinery.

A bigger desk only helps if you can reach everything

GPUs use high-bandwidth memory, or HBM, to hold and move data quickly. Two properties matter: how much it can store and how quickly the processor can access it.

More capacity gives the system room for larger models, longer contexts, or additional requests. But every request also needs bandwidth and compute.

If data can’t move quickly enough to meet users’ response-time expectations, additional capacity won’t automatically translate into useful output.

There’s another complication: everything doesn’t need to stay in the fastest memory all the time.

A paused agent’s context still matters. It just may not need prime desk space while the agent waits for a tool result.

The challenge is deciding what to keep close, what to move elsewhere, and whether retrieving it later will be faster than rebuilding it.

Shrink what needs storing

The first technique is quantization: representing numerical values with fewer bits.

NVIDIA’s NVFP4 format uses four-bit values with additional scaling information to reduce the space needed for model weights. Where the model and serving software support it, lower precision can also shrink the KV cache.

Smaller weights leave more GPU memory available for context and concurrent requests.

But fewer bits come with a question that matters more than the compression ratio: does the model still perform well enough?

As NVIDIA’s video puts it, “we still have to check” performance on the tasks that matter. A system that fits more requests but produces worse answers hasn’t necessarily become more economical.

That’s especially relevant for agents. A small quality loss can become expensive if it causes extra tool calls, retries, or failed tasks.

Spread the model—and account for the traffic

Another approach is to distribute model weights across more GPUs.

Mixture-of-experts models are a useful example. They contain specialized groups of parameters, called experts, but activate only a subset for each token. All those experts still need somewhere to live.

Spreading them across GPUs reduces the share of each GPU’s memory occupied by expert weights. That can leave more room for active context.

The catch is communication. Tokens must reach the GPUs holding the selected experts, and the results must return.

Fast interconnects such as NVLink help make that exchange practical. Each GPU still has its own HBM; distributing the model doesn’t erase the cost of moving information between devices.

The benefit depends on whether the memory and computation gains outweigh the additional traffic.

Stop paying to process the same beginning

Many agent requests share material: system instructions, tool definitions, or the beginning of a document.

Reprocessing those identical prefixes wastes work.

NVIDIA’s Dynamo serving framework helps route requests using information about existing cache state and worker load. When a matching prefix and compatible cache are available, the system can reuse earlier calculations.

Its KV-aware routing documentation describes the practical tradeoff: send work somewhere that already has useful context while also considering how busy that worker is.

That qualification matters. Routing every request to the machine with the best cache match won’t help if that machine becomes a queue.

Reuse is valuable when it improves the whole request’s journey.

Give paused context somewhere cheaper to wait

NVIDIA’s Vera CPU addresses another part of the problem.

Faster CPU execution can shorten tool calls, allowing an agent to resume sooner. As a host CPU alongside Rubin GPUs, Vera also provides nearby system memory that can hold offloaded context when the serving software is configured to use it.

Beyond an individual server, NVIDIA’s CMX adds shared flash storage for inference context. Compatible servers can retrieve cached work rather than independently processing the same input again.

The video makes two limits clear: the cache must match what the model needs, and the requesting server must be allowed to use it.

Moving context also takes time. Offloading pays off when retrieving saved state is preferable to keeping it in HBM or recomputing it within the response-time budget.

This builds on the broader system strategy we covered in NVIDIA’s GTC 2026 announcements: the serving software, CPU, and storage increasingly determine how much useful work the GPUs deliver.

The business question is what an agent actually finishes

NVIDIA attaches a striking number to its argument. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, the company reports up to 30 times higher throughput per dollar for Vera Rubin NVL72 than GB300 NVL72 at 125 tokens per second per user, with roughly 3% more HBM capacity per GPU.

That is a vendor-reported result for a specified workload and operating point. It doesn’t establish that moving context to storage will make an arbitrary application 30 times cheaper.

The architectural lesson is still useful: memory capacity alone is a poor guide to system economics.

For teams evaluating agent infrastructure, the meaningful questions are practical. Does compression preserve task quality? How often can context be reused? Does offloading improve response times under real concurrency? And does the system finish more successful tasks for the money?

There’s a parallel effort happening inside models themselves, as our DeepSeek V4.1 Flash explainer explores. Model architecture and serving infrastructure offer different ways to reduce the burden of carrying context.

The larger signal is that agents are making waiting, remembering, and retrieving central engineering problems.

A successful agent service needs to do more than generate tokens quickly. It needs to carry useful context through an entire job without making every pause an expensive parking spot.

Related reading

• Everything NVIDIA Announced at GTC 2026, Explained — The broader hardware and software strategy behind NVIDIA’s approach to agent infrastructure.
• DeepSeek V4.1 Flash Explained: How It Cuts AI Memory 8x — A complementary look at reducing memory requirements through model architecture.