Performance
The Batch Job That Paid Full Price for One Call at a Time
The nightly job sent each record separately with the same system prompt, so the shared prefix was billed eight thousand times instead of once, and the bill…
Performance
The nightly job sent each record separately with the same system prompt, so the shared prefix was billed eight thousand times instead of once, and the bill…
Performance
Every retrieved document went into every prompt because retrieval was cheap, and the prefill cost of the context nobody read became the latency users waited…