The Batch Job That Paid Full Price for One Call at a Time

I would like to say the two hundred dollars was the problem, because that would be a shorter post, but it started at the thousand.

The repeated prefix billing has a single weak property, and here it is: each call is priced on its own tokens, and the shared prefix is counted in every call, so the cost of the instructions is multiplied by the number of records rather than paid once.

I have since met the same mechanics in plenty of batch that repeats a long prompt.

Why the prefix was billed every time

The call is the unit of billing, and the unit is the request, and the request contains the system prompt and the record, and the contains is the tokens, and the tokens are the price, and the price is the sum, and the sum includes the prefix, and the prefix is the instructions, and the instructions are the same, and the same is the repetition, and the repetition is the eight thousand, and the eight thousand is the multiplication, and the multiplication is the thousand dollars.

The prefix was two thousand tokens, and the record was two hundred, and the two thousand dominated, and the dominated was the cost, and the cost was the prefix, and the prefix was the instructions, and the instructions were necessary, and the necessary was the repetition, and the repetition was the waste, and the waste was the fix's target, and the target was the batching, and the batching was the sharing, and the sharing was the two hundred dollars.

The arithmetic is the argument, because the prefix is ninety percent of the tokens, and the ninety percent is the cost, and the cost is the prefix, and the prefix is the shared, and the shared should be paid once, and the once is the batching, and the batching is the fix, and the fix is the eight hundred dollars saved, and the saved is the audit's finding.

Why nobody noticed

The job worked, and the worked was the signal, and the signal was the pass, and the pass was the cost's invisibility, because the cost is not in the logs, and the not in the logs is the unmeasured, and the unmeasured is the unknown, and the unknown is the surprise, and the surprise is the bill, and the bill is the first measurement, and the first measurement is the late one, and the late is the cost of the unmeasured.

The per-call cost looked small, and the small was the reassurance, and the reassurance was the multiplication, and the multiplication was the eight thousand, and the eight thousand was the thousand dollars, and the thousand was the surprise, and the surprise is the reason the batch cost should be measured as a batch, because the per-call cost hides the repetition, and the repetition is the actual cost.

This is the same misreading as the agent token bill as a performance metric, and the common root is the property, not the platform.

What actually fixed it

Batch the records into one call. The eight thousand records became eighty calls of one hundred records each, and the one hundred is the batch, and the batch is the shared prefix, and the shared is the once, and the once is the saving, and the saving is the eight hundred dollars, and the dollars are the fix, and the fix is the batching, and the batching is the arithmetic, because the prefix is now paid eighty times instead of eight thousand.

Use the provider's batch endpoint. The batch API accepts the whole job and prices it lower, and the lower is the discount, and the discount is the saving, and the saving is the fix, and the fix is the endpoint, and the endpoint is the asynchronous, and the asynchronous is the nightly job's fit, and the fit is the reason the batch API exists, and the exists is the fix.

Enable prompt caching. The provider caches the prefix, and the cached prefix is priced lower on repeat, and the lower is the saving, and the saving is the fix, and the fix is the cache, and the cache requires the prefix to be stable, and the stable is the ordering, and the ordering is the discipline, because the prefix must come first and be identical for the cache to hit.

Shrink the prompt. The instructions were trimmed, and the trimmed is the fewer tokens, and the fewer is the lower cost, and the lower is the saving, and the saving is the fix, and the fix is the edit, and the edit is the review, and the review is the question of what the model actually needs, and the needs is the minimum, and the minimum is the cost's floor.

Measure the batch, not the call. The job reports its total cost, and the total is the metric, and the metric is the alert, and the alert is the detection, and the detection is the prevention, and the prevention is the measurement, and the measurement is the fix, because the unmeasured batch is the surprise bill, and the surprise is the incident.

The rule

Each call is billed on its own tokens, so a repeated system prompt is paid once per record rather than once per job. Batch the records, use the provider's batch endpoint, enable prompt caching with a stable prefix, trim the instructions, and measure the batch total rather than the per-call cost.

The nightly job cost a thousand dollars and eight hundred of it was the same two thousand tokens sent eight thousand times. That is the whole pattern, and it is the part worth remembering.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud