The CAG Bill Nobody Budgets For: Token Economics and the VRAM Wall

Part 2 of 2: When caching actually pays off, the point where it inverts and never recovers, and the hardware reality that decides whether any of this is affordable at scale.

Series — RAG vs CAG: (1) RAG fetches, CAG remembers — how context gets supplied. (2) Token economics and the VRAM wall.

In Part 1, I made caching sound almost too good: skip the search, keep the model's memory warm, turn seconds of latency into milliseconds. Then I hinted that one file save could blow the whole thing up.

This part is about the two forces that decide whether CAG is a smart bet or a slow-motion budget disaster: the token economics and the VRAM wall. Both are counterintuitive. Both have a specific number where the "obvious" choice flips to wrong.

Grab a coffee. This is the part where I show you the math I wish someone had shown me before I turned caching on.

Why Isn't Cached Inference Just "Cheaper"?

Because "cached" isn't one price. It's three.

Ephemeral cache pricing: write premium, read discount

Figure 1: Writing to the cache costs a ~25% premium over base input. Reading from it costs ~90% less. The gap between those two numbers is the entire economic game. (Rates as reported — see Asterisks.)

Providers price cache operations on a tiered card, and the two tiers that matter pull in opposite directions. On Anthropic's Sonnet-class rates, base input runs about $3.00 per million tokens. Writing a new prefix into the cache costs a 25% premium — around $3.75 — because you're paying to build and hold that state. But a query that hits the cache reads at $0.30 — a 90% discount.

So caching is a bet: pay a premium up front, then collect a deep discount on every subsequent read. Whether you come out ahead depends entirely on how many reads you get before the cache goes stale.

Google structures it differently — implicit caching passes savings automatically, but explicit caching charges an hourly storage fee (roughly $1.00 per million tokens per hour on Flash-class models) on top of the read discount. Same idea, different meter: Anthropic charges you a premium to write; Google charges you rent to hold.

That difference isn't trivia. It changes which workloads each provider suits. And it sets up the question every architect eventually has to answer: how many queries before the cache pays for itself?

When Does CAG Actually Pay Off?

There's a clean breakeven, and for a mid-sized repo it's lower than you'd guess.

Breakeven at a 50K-token codebase: about 33 queries

Figure 2: For a stable 50K-token codebase, CAG's expensive first write is amortized after roughly 33 queries. Past that line, discounted reads beat RAG's full-price small payloads.

Model it out. RAG processes a small payload every time — say a 5,000-token retrieval plus your prompt — always at full price. CAG pays the big write once, then reads the whole context cheap on every turn.

For a 50,000-token codebase with a 5K RAG payload, the lines cross at roughly 33 queries. Ask fewer than 33 questions of that stable repo and RAG was cheaper. Ask more, and CAG's discounted reads pull ahead and stay ahead.

If the story ended here, the advice would be simple: "stable repo, lots of queries, turn on caching." But the story does not end here. Because that breakeven quietly depends on codebase size — and when you scale the codebase up, something breaks.

The Inversion Nobody Sees Coming

The scaling inversion: past 100K tokens, CAG never wins

Figure 3: RAG's per-query cost is flat — it only ever reads its small payload. CAG's per-query read cost grows with the whole codebase. Past ~100K tokens, even a 90% discount on everything loses to full price on a little.

Here's the trap. RAG's per-query cost barely moves as the repo grows, because RAG only ever reads its small retrieved payload — 5K tokens whether the repo is 50K or 500K. CAG's per-query read cost, though, scales with the entire cached context. You're reading the whole repo every time, even at 90% off.

Do the arithmetic and it flips. For a 50K codebase, CAG's cheap reads win after breakeven. But as the codebase crosses 100K tokens, the discounted read of everything becomes more expensive than a full-price read of a little. Past that point, CAG is never cheaper per query than RAG — at any session length. More queries don't rescue it. The per-query cost is simply higher, forever.

And it gets worse in multi-turn chat, where conversation history piles onto the cached codebase. Model a 20-turn conversation against a 200K-token codebase and CAG comes out roughly 95% more expensive than a stateful RAG setup — because that giant context plus the growing chat history gets read from cache on every single turn. The read costs compound until they swamp any theoretical saving.

What this means for engineering leaders: CAG's economics are excellent in a narrow band — stable context, moderate size, high query-to-edit ratio — and actively hostile outside it. This is precisely why Kiro's approach is so quietly clever: by caching small, stable Spec-Driven Development artifacts (design.md, tech.md) instead of raw repos, it lives permanently inside the profitable zone and never touches the inversion. The lesson isn't "don't cache." It's "cache the small, stable, high-value thing — not the whole world."

Even If You Can Afford It — Can Your GPU Hold It?

Cost is only half the wall. The other half is physical, and it's less negotiable.

Context overload: the KV cache eclipses model weights by 2.4x

Figure 4: A 70B model in FP8 is ~70 GB of weights. Cache a 128K-token context for just 8 concurrent developers and the KV cache alone is ~171 GB — 2.4x the model itself, and well past a single GPU.

Every token you hold in cache occupies real space in the GPU's High Bandwidth Memory. The KV cache size follows a structural formula — layers times KV heads times head dimension times sequence length times batch size times bytes-per-element — and it grows with everything at once: longer context, more concurrent users, higher precision.

Put numbers on it. A 70-billion-parameter model quantized to FP8 is about 70 GB of weights. Now serve a 128K-token codebase to just 8 concurrent developers under CAG, and the KV cache alone demands roughly 171 GB of VRAM — about 2.4x the model weights. That's "context overload," and it doesn't fit on one H100. When VRAM runs out, the serving system starts evicting cache aggressively, stalling the decode pipeline and forcing expensive re-prefills — the exact thing caching was supposed to prevent.

So the naive CAG dream — "just keep everything warm for everyone" — collides with silicon almost immediately. The industry's answer is a stack of tricks to make the cache fit, and then to spread it across more than one kind of memory.

How Do Serious Deployments Fit the Cache?

Two moves: compress it, then tier it.

Compression comes from PagedAttention — the vLLM/TensorRT-LLM technique that treats the KV cache like OS virtual memory, allocating non-contiguous pages on demand instead of reserving giant contiguous blocks. It cuts memory waste to under 4% and lets a cluster pack far more concurrent users onto the same hardware. On top of that, quantization schemes squeeze the cache further — TurboQuant reportedly compresses to ~3 bits per value (from 16) for a ~6x reduction with negligible accuracy loss, and architectures like Multi-Head Latent Attention cut the cache footprint by up to ~93% versus standard attention (see Asterisks).

Compression buys headroom. It doesn't give you infinite VRAM. So the cache spills across tiers.

Figure 5: Hot state lives in GPU HBM; evicted blocks cascade down through CPU DRAM, local NVMe, and finally shared remote storage. Faster and pricier at the top, bigger and cheaper at the bottom.

Platforms like LMCache extend the cache beyond the GPU into a four-tier hierarchy. Tier 0 is GPU HBM — sub-millisecond, the hot cache for active generation. Tier 1 is CPU DRAM — warm, ~5-microsecond access, holding recently evicted blocks. Tier 2 is local NVMe — cold, 100-500 microseconds, for massive states that outgrow RAM. Tier 3 is remote RDMA or S3 — shared across nodes so a precomputed codebase survives spot-instance restarts and can be reused by every worker in the fleet.

It's a memory hierarchy, the same idea your CPU has used for decades — just applied to attention state instead of cache lines. Evicted blocks fall down the tiers; hot requests pull warm state back up.

Why Split Prefill From Decode At All?

Because they're two different kinds of workload fighting over one machine.

Disaggregated serving: separate compute-bound prefill from memory-bound decode

Figure 6: Prefill is compute-bound (dense matrix math); decode is memory-bound (bandwidth to the KV cache). Run them on the same GPU and a big prefill freezes everyone's decoding. Split them and a cache-aware router keeps sessions pinned to their warm state.

Here's the physical conflict. Prefill is compute-bound — it's dense matrix multiplication, hungry for raw FLOPS. Decode is memory-bound — each generated token just needs bandwidth to load the KV cache. Run both on the same hardware and they sabotage each other: one 100K-token prefill will completely stall token generation for every other user on that box.

The fix is disaggregated serving. Dedicate high-compute GPUs (H100, B200) to prefill. Ship the resulting KV cache over NVLink or InfiniBand to memory-rich decode nodes (an H200 with 141 GB of HBM3e). Then a cache-aware router — llm-d, or a vLLM Router wired through Envoy — pins each incoming session to the specific decode worker already holding its warm cache.

What this means for engineering leaders: by the time CAG is genuinely worth it at scale, you're not choosing a prompt strategy anymore — you're committing to a serving architecture. Disaggregated prefill and decode, cache-aware routing, tiered memory. That's a platform decision with a platform-sized price tag, and it belongs in the same budget conversation as the rest of your inference infrastructure.

So What Should You Actually Build?

Not pure RAG. Not pure CAG. The honest answer the whole industry is converging on is a layered hybrid, and it maps cleanly onto everything from Part 1:

Cache the stable, high-value core. Structured specs and architectural docs — Kiro-style design.md and tech.md — are small, change rarely, and cache beautifully in HBM or DRAM. This is your persistent CAG foundation, permanently on the profitable side of the inversion.
Retrieve the volatile stuff. Active-development files and sprawling multi-repo telemetry go through an MCP-enabled RAG pipeline with AST chunking and cross-encoder reranking to fight Lost-in-the-Middle.
Split the hardware to match. Compute-heavy nodes handle the static cached prefixes; memory-heavy nodes handle the dynamic decodes; cache-aware routing ties it together.

You get holistic reasoning where the context is stable and cheap dynamism where it isn't — without betting the GPU budget on keeping the entire world warm.

The One-Sentence Summary

CAG wins in a narrow band — small, stable, heavily-queried context — and loses everywhere else on both cost and VRAM; the durable architecture caches the stable core, retrieves the volatile edges, and splits the hardware to serve each.

Where This Connects

If you've read the Context Engineering series, this is the retrieval layer sitting underneath it. Part 1 of that series called the context window the new RAM. This companion adds the corollary: RAM has a memory controller, and RAG-vs-CAG is the fetch policy. Choosing it deliberately — per layer, per volatility profile — is what separates a demo from a platform.

Asterisks — verify before you cite

The economic and hardware figures here come from the underlying research brief, provider rate cards, and third-party benchmarks. They're directional and time-sensitive. Confirm against primary sources before repeating:

Pricing (Sonnet-class $3.00 base / $3.75 write / $0.30 read; Gemini hourly storage ~$1.00/MTok/hr; Nova Pro rates) reflects reported rate cards and will drift. Check the provider's current pricing page before quoting.
Breakeven (~33 queries at 50K), the 100K inversion, and the ~95% multi-turn premium come from a specific cost model with fixed assumptions (payload sizes, static codebase, particular rate card). Your numbers will move with your assumptions — treat the shape of the curves as the takeaway, not the exact values.
VRAM figures (~70 GB weights, ~171 GB cache for 128K × 8 users, 2.4x ratio) assume a specific 70B model, FP8 quantization, and batch profile. Recompute for your own model and concurrency.
Compression claims (TurboQuant ~6x, MLA ~93%, PagedAttention <4% waste) are from vendor and research reports, not my independent measurement.
Tool and hardware specifics (H100/H200/B200, llm-d, LMCache tiers, Antigravity Cache IDs) are a mid-2026 snapshot and move quickly.

This is Part 2 of a two-part companion to the Context Engineering series. Toolkit: github.com/navendubrajesh/context-management-for-agents · Writing: medium.com/@navendubrajesh