PublishMDfiles

RAG vs CAG · Part 1 of 2

RAG Fetches. CAG Remembers. The Choice Is Quietly Rewriting Your Context Budget.

Two ways to feed a model your codebase — and why the "obvious" one is wrong more often than you'd think. A companion to the Context Engineering series.

Updated 7/9/2026

RAG Fetches. CAG Remembers. The Choice Is Quietly Rewriting Your Context Budget.

Part 1 of 2: Two ways to feed a model your codebase — and why the "obvious" one is wrong more often than you'd think. A companion to the Context Engineering series.

Series — RAG vs CAG: (1) RAG fetches, CAG remembers — how context gets supplied. (2) Token economics and the VRAM wall.

I spent a chunk of last month watching an agent burn through a 200K-token window on a repo that was only 60K tokens on disk. The math didn't add up until I looked at how the context was being fed in. That's when it clicked: the interesting question in 2026 isn't "how big is your context window." It's "how does anything get into it in the first place."

There are two answers competing for that job. One fetches. One remembers. And the folk wisdom — "just cache everything, windows are huge now" — turns out to be a fast way to set money on fire.

Let me walk you through both, the way I'd sketch it on a whiteboard for a colleague who's about to make an architecture call.

The Two Philosophies, In One Picture

Stateless RAG versus stateful CAG
Figure 1: RAG rebuilds a fresh, minimal context for every query. CAG precomputes the whole repository once and reuses that warm memory on every query. Two completely different bets about statefulness.

Here's the whole debate in one sentence. RAG treats the model as a stateless engine. CAG treats it as a stateful processor.

RAG — Retrieval-Augmented Generation — assumes the model remembers nothing between turns, so it assembles a small, bespoke context payload for each query. Find the relevant bits, inject them, throw them away.

CAG — Cache-Augmented Generation — assumes you'd rather pay once to precompute everything and then reuse it. Load the entire repo into the window, freeze the model's internal state, and reload that frozen state on every subsequent query.

Same goal — get the right context in front of the model. Opposite strategies. And the strategy you pick has real downstream consequences for latency, cost, and whether your agent quietly starts hallucinating around hour three.

How Does RAG Actually Find the Right Code?

RAG is the incumbent. It's what most tools were built on, and it works on a just-in-time premise: don't overwhelm the model, fetch what matters when it matters.

The modern RAG pipeline for code
Figure 2: The pipeline — parse into semantic chunks, embed, store, similarity-search at query time, inject the top hits. The two green boxes are the enterprise-grade refinements that make it actually usable.

The pipeline reads left to right. The repo gets segmented into chunks, each chunk becomes a high-dimensional embedding, and everything lands in a vector database. When you ask a question, the system runs a cosine-similarity search, grabs the closest chunks, and staples them onto your prompt.

The naive version of this splits code by character count, which is how you end up with a chunk that ends mid-function. Modern tools stopped doing that. They parse code into an Abstract Syntax Tree first — using something like tree-sitter — and split at functional boundaries. A function stays whole. A class stays whole. The splits land where a human would put them.

Two more refinements matter, and they're the reason RAG survives contact with a real enterprise:

So RAG is dynamic, cheap to update, and privacy-aware. Sounds great. Then you hit the wall everyone hits.

Why Does RAG Lose the Plot in Long Contexts?

Two problems, and the second is the one that keeps me up at night.

The first is retrieval quality. Vector embeddings compress every document into a fixed-dimensional space before your query ever arrives. That compression loses nuance — especially the cross-file architectural dependencies that don't sit near each other in vector space. And "semantically similar" is not the same as "logically relevant." Ask for a specific auth mechanism and the search may hand you a probabilistically similar but functionally unrelated crypto routine. Close in vector space. Useless in practice.

The second problem is nastier because it's about the model's attention itself.

Lost in the Middle: attention collapses in the center as the window fills
Figure 3: Transformers over-attend to the beginning (primacy) and the end (recency). Past ~50% window fill, primacy collapses and the middle goes effectively invisible.

This is the "Lost in the Middle" phenomenon, and it's not a quirk — it's a structural property of how transformers allocate attention. Models pay disproportionate attention to the very start of the context (primacy bias) and the very end (recency bias). The middle gets shortchanged.

Here's the part that changes how you should think about window sizing. The biases don't scale uniformly. When your input sits under 50% of the window, primacy holds — the model still reliably recalls early instructions. Cross that 50% threshold and primacy degrades hard. Now only recency is really working. Everything you carefully placed at the top of an over-stuffed prompt? Effectively invisible.

Read that again, because it reframes the whole "million-token window" marketing. A bigger tank doesn't help if the model stops reading the middle of it.

RAG's mitigations are real but partial — two-stage retrieval with cross-encoder reranking to push the best chunks to the absolute start and end, positional-encoding tricks like Ms-PoE to preserve mid-range dependencies. They help. They don't solve holistic reasoning, where you genuinely need the whole codebase in view at once. And that's exactly the gap CAG walks into.

How Does CAG Skip the Whole Search Problem?

CAG's pitch is blunt: stop searching. Just load everything.

How CAG works: pay for the prefill once, reuse the KV cache
Figure 4: The prefill computes attention tensors for every token once (slow, expensive). Persist that KV cache, and every later query skips straight to decode — turning seconds of latency into milliseconds.

To see why this works, you need one mechanical detail about how transformers run. During the initial pass over a prompt — the prefill — the model computes Key and Value attention tensors for every single token. In a normal stateless call, all that work gets thrown away the instant the response finishes.

CAG's move is to keep it. Persist the KV cache in fast memory or on disk. When the next query arrives, reload the cached state, append the new query tokens, and jump straight to decoding. You never re-pay for the prefill on the stable part of the context.

The payoff is dramatic on latency. Time To First Token drops from seconds to milliseconds — reported speedups run up to ~56x on 130K-token prompts (see Asterisks). And because the entire codebase is genuinely present in the active cache, the model reasons across files without depending on a search algorithm's guess about what's relevant. No chunking loss. No "close in vector space." The whole thing is just… there.

If that sounds like it dodges every RAG problem at once — it does. Which is exactly why people over-adopt it. Because CAG has its own failure mode, and it's a brutal one.

What Breaks CAG? One File Save.

One file save, two very different bills
Figure 5: Under RAG, a save re-embeds one file — trivial. Under CAG, a save to an early file invalidates everything downstream in the cache, forcing a full re-prefill.

The KV cache is a sequential, mathematically interdependent structure. Each token's tensors depend on everything before it. So when you modify a file, the cached tensors for that file — and every token after it in the sequence — instantly become invalid. This is cache staleness, and it's the whole ballgame.

Now picture a real developer. They're saving files constantly, testing iteratively, editing in tight loops. That environment is the natural enemy of a frozen cache.

Under RAG, a save is a shrug: re-parse one file, re-embed, done. Under a pure CAG setup, a save to a core file blows up the prefix cache. The next query is a guaranteed cache miss, which forces a fresh, GPU-heavy prefill pass over the entire codebase. You pay full freight again.

The deciding factor is the ratio of saves to queries. Edit constantly and query rarely, and CAG makes you re-write massive contexts over and over — far more expensive than RAG's trivial vector updates. Query a stable repo many times without touching it, and CAG's cheap reads win. That ratio is the hinge the entire decision swings on, and I'll put hard numbers on it in Part 2.

So Which One Do the Real Tools Actually Use?

Neither, purely. And that's the honest answer that the RAG-vs-CAG framing tends to hide.

Where the five tools sit on the RAG-to-CAG spectrum
Figure 6: The five tools from the companion series, placed on the spectrum — plus the three hybrid patterns that bridge the gap. (Tool behaviors are a mid-2026 snapshot; verify against current docs — see Asterisks.)

Line the five tools up and you get a spectrum, not a binary:

And the bridges between the two philosophies already have names. CRAG adds a self-correcting evaluator that scores retrieved chunks and re-queries when they're weak. kvRAG retrieves chunks the RAG way but loads their precomputed KV states, skipping prefill for those segments. CacheRAG caches logical query plans instead of raw text.

What this means for engineering leaders: the question is never "RAG or CAG." It's "which parts of my context are stable enough to cache, and which are too volatile to freeze." Get that split wrong and you either pay for constant re-prefills or you fragment the reasoning you were trying to preserve.

The One-Sentence Summary

RAG fetches a little on demand and stays cheap to update; CAG remembers everything and stays cheap to read — and the entire decision comes down to how often your context changes versus how often you query it.

What Comes Next

That's the mechanics. Part 2 is where it gets expensive: the token economics that decide when each approach actually pays off (there's a breakeven point, and then there's an inversion that surprises people), and the hardware reality underneath — the VRAM math, the tiered storage, and why serious deployments physically split prefill from decode.

If Part 1 was "how do these two things work," Part 2 is "what do they cost, and what does it take to run them at scale without the GPU bill eating your margin."

Asterisks — verify before you cite

A few claims here are drawn from the underlying research brief and third-party reports rather than my own first-hand measurement. Treat these as directional and confirm against a primary source before repeating them:

This is Part 1 of a two-part companion to the Context Engineering series. Toolkit: github.com/navendubrajesh/context-management-for-agents · Writing: medium.com/@navendubrajesh