Consumer lag steadily climbs under load, and the consumer group keeps rebalancing, causing message processing to stall intermittently.
Tune poll and session timeout settings, match the number of consumers to available partitions, and move slow processing work off the polling thread.
Step-by-Step Guide
Check lag per partition: kafka-consumer-groups.sh --describe --group my-group
Ensure consumer instances don't exceed the number of partitions, since extras sit idle
Increase max.poll.interval.ms if processing regularly takes longer than the default timeout
Move slow work like DB writes or external API calls off the polling thread into an async pool
Tune fetch.min.bytes and max.poll.records to batch messages more efficiently
Balance session.timeout.ms against heartbeat.interval.ms to avoid false 'consumer dead' detection
Add more partitions to the topic if a single consumer genuinely can't keep up
Make processing idempotent so retries after a rebalance don't duplicate side effects
Found an issue with this solution?
