The Context Stuffing That Inflated Time to First Token

I would like to say the cost of the cheap retrieval was the problem, because that would be a shorter post, but it started at the slow.

This is the anatomy of context stuffing, and of the specific property that makes it a latency problem rather than a cost problem: the prefill cost is paid before the first token, so the unused context delays the response the user is waiting for, and the delay is the perceived latency, and the perceived latency is the abandonment.

This applies to any retrieval-augmented feature, and the mechanism is the prefill, which is the same regardless of the model.

Why the context was full

The retrieval returned the top results, and the top results were ten, and the ten were included because the inclusion was cheap, and the cheap is the embedding lookup, and the lookup is milliseconds, and the milliseconds are the reason the ten were included, and the included ten are the context, and the context is the prompt, and the prompt is the prefill, and the prefill is the seconds, and the seconds are the wait, and the wait is the cost of the cheap retrieval, and the cost is the asymmetry, because the retrieval is milliseconds and the prefill is seconds, and the asymmetry is the reason the cheap retrieval is expensive.

The inclusion was the default, because excluding a candidate requires a decision, and the decision requires a threshold, and the threshold requires tuning, and the tuning is the work nobody scheduled. So the default became include everything, and include everything is the stuffing, and the stuffing is the latency.

The model reads the whole context, and the reading is the prefill, and the prefill is the cost, and the cost is proportional to the length, and the length is the ten documents, and the ten documents are the thousands of tokens, and the thousands of tokens are the seconds, and the seconds are the wait, and the wait is the abandonment, and the abandonment is the cost.

Why the unused context still costs

The model does not skip the irrelevant context, because the attention is over the whole prompt, and the whole prompt is the prefill, and the prefill is the cost, and the cost is paid for every token, and the every token includes the irrelevant, and the irrelevant is the majority, because the retrieval's precision is low, and the low precision means most of the ten are not relevant, and the not relevant are the cost without the benefit, and the cost without the benefit is the waste, and the waste is the latency, and the latency is the wait.

The model may use one document and ignore nine, and the nine are still prefilled, so the nine are the seconds the user waits, and the waiting is the abandonment. The nine are the stuffing, and the stuffing is the reduction's target.

This is the same shape as the agent token bill as a performance metric, same mechanism, different costume.

What actually fixed it

Cut the context to what the model needs. The retrieval's top ten became the top three, and the three are the relevant, and the relevant are the benefit, and the benefit is the answer, and the answer is the feature, and the feature is the same with three as with ten, because the seven were not used, and the not used are the cost without the benefit, and the cut is the fix, and the fix is the latency's reduction, and the reduction is the seconds, and the seconds are the user's wait.

Rerank before including. The candidates are scored by relevance, and the scored are sorted, and the sorted are cut at the threshold, and the threshold is the precision, and the precision is the relevance, and the relevance is the benefit, and the benefit is the inclusion's criterion, and the criterion is the fix, because the inclusion is now earned rather than default, and the earned is the reduction, and the reduction is the latency.

Compress the context. The documents are summarised before inclusion, and the summarised are shorter, and the shorter are the fewer tokens, and the fewer tokens are the faster prefill, and the faster prefill is the lower latency, and the lower latency is the fix, and the fix is the compression, and the compression is the trade, because the summary may lose the detail, and the detail is the risk, and the risk is the reason to compress selectively, and the selective is the tuning.

Cache the prefix. The system prompt and the stable context are cached, and the cached prefix is not re-prefilled, and the not re-prefilled is the saved time, and the saved time is the lower latency, and the lower latency is the fix, and the fix is the cache, and the cache is the provider's feature, and the feature is the control, and the control is the discipline.

Stream the response. The streaming is the perceived latency's fix, because the first token arrives before the whole response, and the first token is the feedback, and the feedback is the wait's end, and the wait's end is the perceived latency, and the perceived latency is the user's experience, and the experience is the metric, and the metric is the streaming's benefit, and the benefit is the fix, and the fix is the streaming.

Measure the prefill separately. Time to first token gets its own metric and its own alert, because an unmeasured latency is an undetected regression, and the detection is the prevention.

The takeaway

The prefill cost is paid before the first token and is proportional to the prompt length, so context that the model does not use still delays the response. Cut the retrieved set, rerank before including, compress selectively, cache the stable prefix, stream the response, and measure time to first token as its own metric.

The feature was slow because the prompt was long, and the prompt was long because the retrieval was cheap, and the cheap retrieval filled the context with documents the model never read, and the unread documents were the seconds the user waited. I now check for that property first, because it is where the failure actually lives.

Related reading: The Batch Job That Paid Full Price for One Call at a Time explores a connected angle.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud