kafka超时异常_kafka maximum size of batches returned in poll(wit

kafka超时异常

Kafka Handle Error, Client Will Seek Soon: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
1

因为一次性poll拉取（默认500）消息后处理时间过长，导致两次拉取时间间隔超过了max.poll.interval.ms阈值（默认五分钟）,集群以为消费线程挂了，触发了rebanlance。当消费完了再去同步游标报错了，然后游标回滚，导致部分重复消费。

消费者参数

6个参数是3对，通俗理解如下：

​ 1，2配合使用，告诉kafka集群，我消费者的处理能力，每秒至少能消费掉

​ 3，4配合使用，告诉kafka集群，在我没事情干的时候，多久尝试拉取一次数据，即使此时没有数据（所以要处理空消息）

​ 5，6配合使用，告诉kafka集群，什么情况你可以认为整个消费者挂了，触发rebanlance

【max.poll.interval.ms和session.timeout.ms区别】

​ KIP-62前只有session.timeout.ms参数

KIP-62后不通过poll()方法发送心跳，而是后台一个心跳线程，这就允许单次poll处理更长时间。不会因为单次处理超时假死引发不必要的rebanlance

max.poll.interval.ms 检测消费者处理线程死亡

session.timeout.ms 检测整个消费者死亡