Consumer Lag Is Not a Throughput Problem Until You Prove It Is

Consumer lag is the number of messages produced to a partition that a consumer group has not yet processed. It is the single most useful metric a queue gives you, and it is routinely misread.

The reflex when lag climbs is to add consumers. That works for exactly one cause and does nothing for the other five, and in two of them it makes matters worse.

Read the shape before you act

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processor

TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID
orders     0          1048221         1048245         24     consumer-1
orders     1          1048190         1048233         43     consumer-2
orders     2          981004          1048250         67246  consumer-3
orders     3          1048201         1048240         39     consumer-4

That is the diagnostic. Three partitions healthy, one 67,000 behind. Adding consumers will not help, because a partition is consumed by exactly one member of a group. Six consumers on four partitions leaves two idle.

Tested on Kafka 3.9, Linux 6.8.

The first question is always the same: is lag spread across all partitions, or concentrated on one?

Cause one: partition skew from the key

Kafka routes by hash(key) % partitions. If your key is customer ID and one customer is fifty times larger than the rest, that customer's partition carries fifty times the traffic and no amount of scaling fixes it.

kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic orders --time -1

Compare partition sizes. Uniform means the key is well distributed. One partition at ten times the others means you found it.

The fixes are all structural. Use a composite key such as customerId:orderId if strict per customer ordering is not required. Or route the outlier explicitly to its own partition and give it dedicated capacity. Repartitioning a live topic is disruptive, so this is worth getting right at design time.

The ordering guarantee is the reason people accept skew: Kafka only guarantees ordering within a partition. Widening the key gives up per customer ordering. Decide whether you actually need it, because most systems assume they do and only genuinely need ordering per entity, not per tenant.

Cause two: one message that will not process

Lag frozen at a precise number, with the consumer alive and logging, means the same message is failing and being retried forever.

CURRENT-OFFSET  981004    (unchanged for 40 minutes)
LAG             67246     (growing only because the producer continues)

The offset not advancing is the signal. The consumer is not slow, it is stuck.

Find the message:

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --partition 2 --offset 981004 --max-messages 1

Usually it is a schema change, a null field the code does not expect, or a payload larger than the consumer will accept. The fix is a dead letter topic and a bounded retry, so a single bad message parks itself instead of halting a partition.

Without a dead letter path, the operational remedy is skipping the offset, which means deliberately losing a message. Make that a conscious decision with a record of what was skipped.

Cause three: rebalancing more than processing

Every time group membership changes, Kafka reassigns partitions and consumption stops during the rebalance. A group that rebalances constantly spends its time coordinating.

grep -c "Revoking previously assigned partitions" consumer.log

Two causes dominate:

Processing takes longer than max.poll.interval.ms. If your handler takes six minutes and the interval is five, the broker concludes the consumer is dead, removes it, and rebalances. The consumer then finishes and rejoins, triggering another rebalance. This loops indefinitely.

max.poll.interval.ms=300000
max.poll.records=100

Reducing max.poll.records is usually the better lever. Fetching 500 records and processing each for a second exceeds any sane interval; fetching 50 does not.

Liveness confusion. session.timeout.ms governs heartbeats, sent on a background thread. max.poll.interval.ms governs progress. A consumer can heartbeat happily while making no progress at all, which is why the second setting exists. Watch both.

Use cooperative rebalancing so unaffected partitions keep flowing:

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Cause four: the consumer is waiting on something else

Most consumers that look slow are not slow at all. They are blocked on a downstream call.

If your handler does a database write per message and the database is at its connection limit, lag is a symptom of connection pool exhaustion, not of Kafka. Adding consumers increases pressure on the pool and makes it worse.

Instrument the handler itself:

consumer.handler.duration          histogram
consumer.handler.db.duration       histogram
consumer.handler.http.duration     histogram

If the outer duration is 400 ms and the database call is 380 ms of it, you have a database problem wearing a queue costume. This is the same reasoning as checking whether a service is slow or its dependency is slow before you scale anything.

Cause five: genuinely not enough consumers

This is the case where scaling works, and you should only conclude it after eliminating the others.

The evidence is lag even across all partitions, handler duration flat and reasonable, downstream calls healthy, and consumer count below partition count. Then add consumers, up to the partition count and no further.

Beyond that you must add partitions, which changes key distribution and cannot be undone. Size topics with headroom at creation for this reason.

What to alert on

Absolute lag is a poor alert. A thousand messages behind on a topic doing 50,000 per second is nothing; on a topic doing ten per second it is hours.

Alert on time lag rather than message lag:

estimated_delay = consumer_lag / current_throughput

Then page on delay exceeding a threshold that reflects your actual requirement. Also alert on offsets not advancing while lag is non zero, because that is the stuck partition case and it is invisible to a threshold on lag alone.

Prevention

• Alert on time behind, not messages behind.
• Give every consumer a dead letter topic and a bounded retry from day one.
• Tune max.poll.records before you tune anything else. It is the setting that most often causes rebalance loops.
• Instrument handler duration separately from downstream calls, so a slow dependency does not read as slow consumption.
• Check partition skew when you design the key, not when lag appears. Repartitioning a busy topic under pressure is not a pleasant afternoon.