The Rate Limit That Counted Every User as One

The rejected behaved. The exhausted behaved. The chain between them is where working as designed stopped meaning working.

The rate limit keyed on the wrong identity has a single weak property, and here it is: every user behind the proxy shares one bucket, so the honest traffic collides, and the attacker behind a different address is unlimited.

This was an API behind a load balancer, and the mechanics are the same for any limiter that reads the connection's source address in a proxied deployment.

Why the key was wrong

The limiter keyed the remote address, and the remote address was the TCP connection's peer, and the peer was the load balancer, and the load balancer was the every request's source, and the source was the one key, and the one key was the shared bucket, and the shared bucket was the collision, and the collision was the 429.

The client's address was the header, and the header was the X-Forwarded-For, and the header was the proxy's added, and the added was the available, and the available was the not read, and the not read was the bug, and the bug was the default, and the default was the connection's peer, and the peer was the wrong identity.

The header is the trusted only from the known proxy, and the known proxy is the allowlist, and the allowlist is the spoofing's prevention, and the prevention is the discipline, because the client can send the header too, and the sent header is the lie, and the lie is the limit's bypass, and the bypass is the incident.

Why the limit was also too weak

The shared bucket was the strict for the honest, and the honest was the office, and the office was the fifty users, and the fifty was the one bucket, and the one bucket was the exhausted, and the exhausted was the 429, and the 429 was the complaint, and the complaint was the incident.

The attacker was the different address, and the different address was the own bucket, and the own bucket was the unlimited relative to the honest, and the unlimited was the abuse, and the abuse was the protection's failure, and the failure was the key's fault, and the fault was the identity, and the identity is the whole design.

The rate limit is the identity's policy, and the policy is the per identity, and the per identity is the key, and the key is the design, and the design is the fix's target, and the target is the right identity, and the right identity is the user or the API key, and the API key is the authenticated, and the authenticated is the reliable, and the reliable is the fix.

This is the same proxy confusion as the CORS configuration that echoed the request's origin, and the common root is the property, not the platform.

The fix, in order

Keyed the limit on the authenticated identity where available. The API key was the bucket, and the bucket was the per user, and the per user was the fair, and the fair was the fix, and the fix was the identity, and the identity was the discipline, because the authenticated identity cannot be spoofed the way the header can.

Read the client IP from the trusted proxy header only. The header was the parsed, and the parsed was the allowlisted proxy's, and the allowlisted was the spoofing's prevention, and the prevention was the fix, and the fix was the configuration, and the configuration was the discipline, because the unvalidated header is the attacker's input.

Used the anonymous bucket with the IP as the fallback. The anonymous was the IP, and the IP was the pre-authentication, and the pre-authentication was the fallback, and the fallback was the fix, and the fix was the two tiers, and the tiers were the discipline, because the unauthenticated request has no better identity.

Set the proxy's real IP handling in the framework. The framework's setting was the trusted proxies, and the proxies were the list, and the list was the correct remote address, and the address was the fix, and the fix was the one configuration, and the configuration was the discipline, because the framework's default is the connection's peer.

Alerted on the 429 rate by identity. The metric was the per-key rejections, and the rejections were the pattern, and the pattern was the shared bucket's detection, and the detection was the fix, and the fix was the monitoring, and the monitoring was the discipline, because the 429 spike on the proxy's address is the symptom.

The rule I keep

A rate limiter behind a proxy must not key on the connection's source address, because that address is the proxy's. Key on the authenticated identity, read the client IP only from a trusted proxy's forwarded header, use the IP as the fallback for anonymous traffic, and alert on the rejection rate per key.

The whole office hit the rate limit while a single attacker was unlimited, because the limiter keyed on the TCP peer address and every request arrived from the load balancer. Name that one property in review and this class of bug gets hard to ship.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud