> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Reviewing the Loop That Fired Ten Thousand Model Calls at Once - URL: https://debugly.dev/reviewing-unbounded-model-concurrency/ - Published: 2026-10-09T14:43:00.000Z - Updated: 2026-10-17T05:56:14.000Z - Description: The batch loop awaited every item concurrently with no semaphore, so the first burst rate limited itself, and the retry storm that followed turned a… - Author: Rohit Bhadani - Tags: Code Review, AI, Python It ended at the loop. It started at the one line. Every hand-off in between did exactly what it was told, and this is the one that lied. I have hit this enough times that I now check for it before I check for anything clever. Here is the shape of the unbounded concurrent model call, and the one property that makes it pass review: the code is correct, the await is right, the gather is idiomatic, and the only missing thing is a limit, and the missing limit is invisible because the code reads as the standard pattern. This was Python asyncio, and the mechanics are the same in any language where concurrent calls are cheap to start and expensive to sustain. ## What the review saw The review saw a batch job, and the batch job mapped a function over a list, and the mapping was concurrent, and the concurrent was the asyncio.gather, and the gather was the idiom, and the idiom was the pass, because the idiom is correct, and the correct is the review's criterion, and the criterion was met. The review did not see the rate, because the rate is not in the code, and the rate is the behaviour, and the behaviour is the ten thousand at once, and the at once is the burst, and the burst is the property the review should check, and the check is the question the code does not answer, which is how many of these run at the same time, and the answer is all of them, and the all is the bug, and the bug is the missing bound. The missing bound is the review's blind spot, because the bound is absent, and the absent is not a line to read, and the not a line is the invisible, and the invisible is the pass, and the pass is the incident, and the incident is the reason the review should ask what is not there. ## Why the retry made it worse The 429 is the provider saying slow down, and the slow down is the instruction, and the instruction has a header, and the header is the retry after, and the retry after is the wait, and the wait is the correct response, and the correct response was not implemented, and the not implemented was the immediate retry, and the immediate was the storm, and the storm was the sustained 429, and the sustained was the account level throttle, and the throttle was the other services, and the other services were the outage, and the outage was the blast radius, and the radius is the reason the retry policy is part of the concurrency design. The immediate retry converts a rate limit into a denial of service, and the denial is self inflicted, and the self inflicted is the embarrassment, and the embarrassment is the incident review, and the review is the fix, and the fix is the backoff, and the backoff is the respect for the header, and the respect is the discipline from [the agent retry loop that rate limited a partner API](https://debugly.dev/agent-retry-loop-rate-limited/), and the common root is the property, not the platform. ## The questions that catch it **How many run at once?** The answer is the concurrency, and the concurrency should be a number, and the number should be explicit, and the explicit is the semaphore, and the semaphore is the bound, and the bound is the fix, and the question is the review's first, because the unbounded is the bug. **What happens on 429?** The answer is the retry policy, and the policy should honour the retry after, and the honour is the backoff, and the backoff is the exponential, and the exponential is the jitter, and the jitter is the decorrelation, and the decorrelation is the storm's absence, and the absence is the fix. **What is the total time?** The answer is the batch's duration, and the duration is the rate times the count, and the rate is the concurrency, and the concurrency is the bound, so the duration is predictable, and the predictable is the plan, and the plan is the review's check, because the unbounded loop's duration is unknown until it fails. **What does the provider see?** The answer is the traffic, and the traffic is the burst, and the burst is the shape the provider throttles, and the throttles is the 429, and the 429 is the answer to the question, so the question is the review's, and the review is the prevention. ## The fix **Bounded the concurrency with a semaphore.** The gather became a bounded pool, and the pool's size is the provider's rate limit, and the limit is the bound, and the bound is the fix, and the fix is the one that turns the burst into a steady rate, and the steady is the sustainable, and the sustainable is the completion, and the completion is the backfill done. **Honoured the retry after header.** The retry waits for the header's duration, and the duration is the provider's instruction, and the instruction is the respect, and the respect is the backoff, and the backoff is the exponential with jitter, and the jitter is the decorrelation, and the decorrelation is the storm's absence. **Made the failure partial.** The job collects the failures and continues, and the continues is the resilience, and the resilience is the partial success, and the partial is the retry later, and the later is the queue, and the queue is the dead letter, and the dead letter is the visibility, and the visibility is the fix, because the all or nothing gather turns one failure into a lost batch. **Reported progress.** The job logs the completed count, and the count is the progress, and the progress is the observability, and the observability is the diagnosis, and the diagnosis is the rate, and the rate is the tuning, and the tuning is the improvement, and the improvement is the fix, because the unmeasured batch cannot be optimised. ## The takeaway An unbounded gather over a large list sends every request at once, which self rate limits, and an immediate retry converts the limit into a storm. Bound the concurrency to the provider's rate, honour the retry after header, make failures partial, and report progress. The backfill was one line and the line was ten thousand simultaneous requests, and the provider throttled the account and the throttle took down the other services. That is the whole pattern, and it is the part worth remembering. --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)