RAG in production: what changes once there's an SLA

Putting a RAG together today is an afternoon's work: LangChain, a Postgres database with pgvector, an embedding model, and that's it, the chat answers citing your documents. The demo works. It always works.

What separates that demo from a production system is three numbers the demo never forces you to look at: p95 latency, cost per query, and how much of the answer actually came from the retrieved context.

The numbers below come from an internal assistant that answers based on the product rules of a platform in production. I don't name the client or quote its figures — what matters here are the orders of magnitude and the decisions.

Latency is a budget, not luck

When someone promises "an answer in under 2 seconds", that stops being a metric and becomes a budget to be split across the stages:

Stage	p95 budget
Embed the question	40 ms
Search the vector index	120 ms
Rerank the passages	80 ms
Generate the answer	1,400 ms

If the sum blows past the SLA, swapping the model won't help: the architecture is what's wrong. And the stage almost everybody underestimates is the search.

In pgvector, choosing the index is the first lever. HNSW gives a better speed-to-recall ratio, but takes longer to build and uses more memory. IVFFlat builds fast and stays small, with worse recall for the same search time. In production, in practice, HNSW wins almost every time — the cost is building the index, which you pay once.

The knob that controls the trade-off sits at query time: hnsw.ef_search (default 40) on HNSW and ivfflat.probes (default 1) on IVFFlat. Raising these values improves recall and costs latency. That, and not the model, is where you buy search quality.

-- HNSW index by cosine similarity
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

-- recall vs latency: tune per query, not across the whole database
SET LOCAL hnsw.ef_search = 100;

SELECT id, content
FROM chunks
WHERE tenant_id = $1 AND doc_type = 'rule'
ORDER BY embedding <=> $2
LIMIT 3;

Look at the WHERE in that query, because it holds the most expensive trap in pgvector: with an approximate index, the filter is applied after the index scan. You ask for 3 results, the index returns the nearest neighbors, the filter throws most of them out and 1 is left — or none. The answer comes back empty and looks like a model bug. The way out is iterative scan (hnsw.iterative_scan), which makes the database fetch more candidates until it fills the LIMIT. Finding this out in production, with a customer asking why the assistant "forgot" the rule, is expensive.

A token is context, and context is a choice

The standard tutorial sends the 10 nearest passages to the model. It's the biggest silent waste in a RAG: you pay input for 10 passages, the model reads 10 passages, and the answer comes out of 2.

In the assistant I measured, the corpus is small and well bounded: 16 rule documents turn into 384 chunks, around 530,000 characters, somewhere near 133,000 tokens. Each question carries 3 passages, roughly 1,400 tokens of context, plus the embedded question. Indexing the whole thing, on an embedding model at $0.02 per million tokens, costs less than a cent. And reindexing touches only what changed, because the content hash decides what is new.

Three decisions keep that number small without losing the answer:

• Chunks with a semantic boundary. Cutting every thousand characters is simple and breaks a table, a list or a rule in half. Cutting by section costs a day of work and improves recall and the answer at the same time.
• Rerank before sending. Pulling 20 candidates from the search and sending 3 to the model is cheaper and more accurate than sending 10 straight through.
• Metadata as a filter, not as text. Tenant, document type and version go in the WHERE, not in the prompt.

The gain from RAG isn't savings, it's being right with a source

There's a common sales argument: "RAG saves tokens because you don't have to send the whole manual in the prompt". The math looks good — sending 133,000 tokens with every question, at a typical price of $3 per million input tokens, would come to about $0.40 per question.

Except that isn't the real alternative. Nobody was going to send the whole manual with every question. Without RAG, the alternative is the assistant not knowing the answer — or worse, inventing a rule that sounds plausible.

The gain from RAG is something else: being right with a cited source. An answer that points to which document and which passage it came from can be checked by whoever received it. That is what makes it possible to run the assistant on top of business rules, pricing and policy. Saving tokens is a consequence of well-chosen context, not the goal.

The money is almost never where you look for it

When I measured the cost of that platform, RAG wasn't the problem. Indexing costs cents, and the context per question is small.

What showed up was something else: across tens of thousands of model calls, prompt caching was switched off — the cache-read token counter was zero on every single one. The most expensive feature, responsible for most of the bill, sends around 8,500 input tokens per call, and a good part of that is a stable prefix: safety rules, system instructions, offer data.

A cache read costs around 10% of the normal input price (writing costs about 25% more than ordinary input, once). On an item that dominates the bill and repeats the same prefix thousands of times a day, the order of magnitude of the savings is hundreds of dollars a month.

Three details decide whether this works:

• The cache is prefix-based. One byte changing at the start invalidates everything that comes after it. A timestamp in the system prompt, JSON with keys in random order and a tool list that changes order all bring down the entire cache.
• There is a minimum size. The prefix needs between 512 and 4,096 tokens, depending on the model. Below that it simply isn't cached, silently.
• You can check. The cache-read token field in the response answers in one minute what an architecture discussion takes a week to speculate about.

The lesson I take from it: before promising savings, measure which slice of the prompt is actually fixed. The honest number comes from measurement, not from an estimate.

There is one more observability trap worth writing down. Embedding spend may not show up on your AI cost dashboard, because the logging is usually a middleware that wraps the language model — and the embedding provider doesn't go through it. The cost is small, but the dashboard ends up lying, and a dashboard that lies is worse than a dashboard that doesn't exist — the same pattern as the controls nobody tested.

What I measure before saying it's ready

1. A latency budget per stage, with p95 measured in production, not as a local average.
2. Search recall on a set of real questions, before blaming the model for the bad answer.
3. Tokens per answer, separating the fixed prefix from the retrieved context.
4. Cache read rate, which needs to be greater than zero.
5. How much of the answer is held up by the context, with the source passage cited and visible to whoever is reading.
6. A deterministic fallback path for when the model is down — even if it's a search over explicit patterns, answering less and getting less wrong.

None of this shows up in the demo. All of it shows up in the first week of production.

Further reading

• This is what we do in AI Systems: a latency budget, evaluation against real questions and cost measured per feature.
• The same discipline, applied to the other side of the product: Chattie's inbox, from 6.7 MB to 19 KB.

Sources

• pgvector documentation — index types, distance operators and how filtering behaves with an approximate index.
• Anthropic's prompt caching documentation — read and write pricing, minimum prefix size and invalidation.
• My step-by-step of the implementation with LangChain and pgvector, on my personal site.