The Token Budget Kill Switch That Never Fired

I would like to say the bill was the problem, because that would be a shorter post, but it started at the surprise.

The guard that cannot fire has a single weak property, and here it is: the guard is present, tested, and armed, so its presence is the assurance, and the assurance is the reason nobody checked whether it could ever trigger.

The case was a Python service calling a hosted model; the mechanics generalise to any budget guard that mixes units.

Why the switch never fired

The guard read the token count from the response, and the token count was a number, and the threshold was a number, and the threshold was set to the monthly budget in dollars, and the dollars were a number, so the comparison compiled, and the comparison ran, and the comparison was tokens against dollars, and the two were different quantities, and the different quantities were never equal in the way the guard needed, so the guard never fired.

The specific failure was the magnitude, because the token count for a month was in the tens of millions, and the budget in dollars was in the thousands, and the guard checked whether the tokens exceeded the dollars, which was true from the first day, and the true should have fired the switch, and the switch did not fire, and the not firing was the second bug, and the second bug was the alert, and the alert was wired to the wrong channel, and the wrong channel was unread, and the unread was the silence.

So there were two failures, and the two were independent, and the independence is why the guard felt safe, because the first failure was the units and the second was the routing, and either one alone would have been caught, and the two together were the silent bill, and the silent bill is the reason to test the guard by firing it.

Why the units mismatch survived review

The review saw a budget guard, and the guard had a threshold, and the threshold had a value, and the value looked reasonable, and the reasonable was the pass, because the reviewer read the intent, and the intent was correct, and the intent is what the review checks, and the units are what the code does, and the code's units are not in the intent, so the review checked the wrong thing.

The variable names made it worse, because the threshold was called budget and the counter was called usage, and the budget and the usage sound like the same kind of quantity, and the same kind is the assumption, and the assumption is the mismatch, because the usage was tokens and the budget was dollars, and the names hid the difference, and the hidden difference is the bug, and the bug is the reason to put the unit in the name.

This is the same silence as the promise chain that swallowed a rejection and reported success, and the shared property is the one worth fixing.

What actually fixed it

Convert to one unit and name it. The counter became a cost in dollars, computed from the token counts and the published prices, and the threshold became a cost in dollars, and the comparison is now dollars against dollars, and the same unit is the fix, and the fix is the naming, because the variables are now cost_usd and budget_usd, and the unit in the name is the guard against the next mismatch, and the guard is the prevention.

Test the guard by firing it. The test injects a usage above the threshold and asserts the switch trips, and the assertion is the proof, and the proof is the fix, because a guard that has never fired is a guard whose behaviour is unknown, and the unknown is the risk, and the risk is removed by the test, and the test is the one that would have caught the units, because the injected usage would not have tripped the switch, and the not tripping is the failure, and the failure is the catch.

Route the alert and verify the route. The alert goes to a channel someone reads, and the route is verified by the same test, because the test asserts the notification was sent and received, and the received is the proof, and the proof is the fix, and the fix is the routing, and the routing is the second failure, and the second failure is closed by the verification.

Fail closed on the guard's own error. The guard raises if it cannot compute the cost, and the raise is the safe default, because a guard that errors silently is a guard that is absent, and the absent is the risk, and the risk is removed by the raise, and the raise is the fail closed, and the fail closed is the discipline, because the guard's failure should be louder than the thing it guards.

Alert on the trend, not only the threshold. The daily cost is charted, and the chart has an anomaly alert, and the anomaly is the early warning, and the early warning is the catch before the threshold, and the catch is the prevention, because the threshold is the last line and the trend is the first, and the first is where the bill is still small.

What I now do

A budget guard that compares different units cannot fire, and a guard that has never fired has unknown behaviour, and an alert routed to an unread channel is an absent alert. Convert everything to one unit and put the unit in the name, test the guard by firing it, verify the alert route, fail closed on the guard's own error, and alert on the trend.

The switch was armed and the bill arrived anyway, because the guard was comparing tokens to dollars and the alert was going nowhere, and both failures were silent. The lesson I keep is the property, not the incident.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud