> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Your RAG Is Slow Because Retrieval Is an Untuned Database - URL: https://debugly.dev/rag-retrieval-is-an-untuned-database/ - Published: 2026-10-04T15:30:00.000Z - Updated: 2026-10-17T05:57:14.000Z - Description: The LLM answered in four hundred milliseconds and the request took four seconds. The gap was retrieval, a vector search over an index nobody sized, chunked… - Author: Rohit Bhadani - Tags: Performance, AI, Databases The team profiled the model and found it fast, and the complaint stood, because the model was never the slow part. The request spent three and a half seconds in retrieval: embedding the query, searching a vector index that was sized by default, fetching chunks, and re ranking them. Retrieval is a database, with a query planner of sorts, an index, a storage layout and a working set, and it had received none of the tuning a database gets, because it was filed under "AI" instead of "data". This is the performance case for treating retrieval as the database it is, with the same questions you would ask Postgres, per the site's long argument that plans and indexes are decisions, not defaults. This was a retrieval stack over a few million chunks with a hosted embedding model, and the fixes below are the standard database moves, translated. ## The partition: embed, search, fetch, rerank The first move is the curl timing template's cousin: measure each stage, because the four stages have four different owners and four different fixes. Log per stage latency per request, per [tracing an LLM pipeline with the observability you already have](https://debugly.dev/tracing-llm-pipelines-minimal-observability/), and partition the p99 by stage, because the average hides the stage that bites, per [p99 and why averages lie](https://debugly.dev/p99-latency-and-why-averages-lie/). In our case the embedding was a tenth, the vector search two seconds, the chunk fetch a second over a cold object store, and the rerank a few hundred milliseconds. The search and the fetch were the database, and both were untuned. ## The index is the query plan A vector index is a trade between recall and speed, configured by parameters the defaults set blindly, and the choice of index type is the schema decision. A flat brute force search is exact and slow, an approximate index is fast and lossy, and the right parameters depend on the corpus size and the recall the product actually needs, which nobody had measured. The tuning loop is the database loop: define the recall you need at the rank you care about, then tune the index parameters against a query sample until you reach it at the latency you want, and record the trade as a decision. Most teams discover they need far less recall than the default assumes, because the reranker and the model tolerate near misses, and giving up a little recall buys a large latency win, which is the same shape as accepting a sequential scan when the selectivity says so, in [the query that was fast until the table grew](https://debugly.dev/query-fast-until-the-table-grew/). ## Chunking is the schema The chunk size and overlap are the table design, and the default chunking produced chunks that were too large to retrieve precisely and too small to carry context, which forced the pipeline to fetch many chunks per query, multiplying the fetch cost. Chunking tuned against the eval set, measuring retrieval precision at k, is schema tuning, and it changes both the quality and the cost, because fewer, better chunks means a smaller k and a shorter context, which also speeds the model. The overlap and the boundary policy are the normalisation decisions, and they deserve the same eval driven review as any schema change, not a constant copied from a tutorial. ## The fetch is IO, tune it like IO The chunk fetch went to a cold object store per chunk, per request, which is the N plus one in [two hundred Redis calls and every one was a round trip](https://debugly.dev/redis-round-trips-pipelining-latency/), reborn with HTTP GETs. The fixes are the database fixes: batch the fetches, cache the hot chunks in memory, and store the chunks with the vectors so the search returns the payload in one round trip. The cache hit rate on chunks, like any cache, is the metric, and the hot set for retrieval is usually small, because queries cluster, so a modest cache removes most of the fetch latency. ## The working set and the growth The index, like any index, lives in memory or it does not, and the corpus had grown past the provisioned memory, so the search paged, which is the threshold crossing in [the query that was fast until the table grew](https://debugly.dev/query-fast-until-the-table-grew/), with the same signature: no code change, a slow week, and a size that crossed a line. The fix is the capacity decision: size the memory to the index's working set, or shard, or prune, and graph the index size against the memory so the next crossing is an alert, not an incident. ## The reranker is the second query The cross encoder rerank is a second, expensive query over the candidates, and it is often applied to too many candidates because the first stage was imprecise. Tuning the first stage to be precise lets you rerank fewer candidates, and the rerank depth is a budget with a quality curve, tuned against the eval like every other number. The two stages are one planner, and optimising them separately is optimising two tables without a join strategy. ## What I now do Retrieval is a database: the index is the query plan, chunking is the schema, the fetch is IO and the working set is capacity, and each has a tuning loop the defaults never ran. Measure the stages, tune recall against a measured need, cache the hot chunks, batch the fetches, size the memory to the index, and treat the rerank as part of one plan. The model was never your latency problem. The database you did not know you had was, and it has been waiting, patiently, for the tuning every database gets. The pleasing part is that the fixes compound. Better chunking raises retrieval precision, which lowers k, which shrinks the fetch batch and the context, which speeds the model and cuts its cost, so one schema decision pays across three stages. That compounding is the signature of a well tuned retrieval stack, and the reason the first hour spent on chunking and the index returns more than a week spent profiling the model. --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)