Hisham AlhussainEvery time a model ships with a bigger context window, someone declares RAG dead. Gemini 2.5 Pro went GA last month with a million tokens of context, and sure enough, the hot takes showed up right on schedule.
I get the appeal. Why build retrieval when you can just paste everything in? But after a couple of years building assistants on top of real enterprise data, I can tell you RAG isn't going anywhere. What's changing is that the hard part was never the retrieval trick. It's the guardrails around it.
Long context doesn't replace retrieval
A few reasons I still retrieve, even with huge windows:
- Cost and latency. Stuffing a million tokens into every request is slow and expensive. Most questions need a few pages, not the whole library.
- Access control. In an enterprise, not everyone can see every document. Retrieval is where you enforce that. You can't "un-paste" a confidential file from a prompt.
- Freshness. Data changes. An index you can update is a lot easier to manage than rebuilding giant prompts.
Long context is great as a complement, though. I'll happily retrieve generously and let the model read more around each hit.
Measure retrieval on its own
The biggest mistake I see is only judging the final answer. If the answer is wrong, you have no idea whether retrieval failed or the model did. So I track retrieval separately:
- Hit rate at top-k. Did the right document show up in the results at all?
- MRR. How high up did it show up?
- Per query type. "What's the limit for X?" and "summarize everything about Y" behave very differently. Break your metrics down.
Chunk the way your experts think
Fixed-size chunks are fine for a demo. In production, I try to mirror how subject matter experts actually think about the content. If a compliance team thinks in terms of permits, conditions, and sites, then the chunks and metadata should follow that structure. The jump in quality from this alone is often bigger than any embedding model upgrade.
Golden sets and regression tests
I keep a set of real questions with known good answers and run it on every change: new chunking, new embedding model, new prompt, new LLM. If the numbers drop, the change doesn't ship. It's simple and it has saved me more than once.
Have a plan for when retrieval is weak
This is the guardrail people skip. When the top results have low similarity, or nothing relevant comes back, the model should say so instead of improvising. A clear "I couldn't find this in the documents, here's who to ask" builds far more trust than a confident wrong answer.
RAG in production is honestly pretty unglamorous. But that's where the trust comes from, and trust is the whole game with enterprise users.