当前位置:   article > 正文

kafka消费信息时,产生重复消费的情况 - 20190121

attempt to join group test failed due to unknown member id, resetting and re

Kafka相关问题:

  1. Marking
  2. the coordinator 2147483283 dead.
  3. Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test Auto offset commit failed: Commit cannot be completed due to group rebalance
  4. Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test Auto offset commit failed:
  5. Attempt to join group test failed due to unknown member id, resetting and retrying.
  6. Attempt to heart beat failed since the group is rebalancing, try to re-join group.we

问题描述:kafka消费信息时,产生重复消费的情况。

问题原因:

大概意思是,在kafka超时时间内,有消息没有处理完成,consumer coordinator会因为没有收到心跳而挂掉,然后自动提交offset失败,然后重新分配partition给客户端。导致以下问题:

1、由于自动提交offset失败,导致重新分配了partition的客户端又重新消费之前的一批数据;

2、接着consumer重新消费,又出现了消费超时,无限循环下去。

消费者消费不过来,数据信息量大,自动commit超时。

设置:max.partition.fetch.bytes:一次拉取最大byte这个属性 小一点.默认1M;

或者设置session.timeout.ms:超时时间设置大一点,在不设置的情况下,kafka默认自动5000ms提交

Ø 用自己的话说,就是kafka中存储了大量数据消费不过来,此时又到了心跳监听阶段,无发送心跳,导致异常。所以要尽量减少数据量,使到监听阶段能够消费完队列中的数据。

资料网站:

https://blog.csdn.net/gpdsjqws/article/details/81027415

【kafka : Error
UNKNOWN_MEMBER_ID occurred while committing offsets for group alert】https://blog.csdn.net/jinzhencs/article/details/71713707

【kafka 0.9.0.0 部分配置详解】https://blog.csdn.net/u012191627/article/details/53338503

【apache kafka 遇到 Attempt to heart beat failed since the group is rebalancing, try to
re-join group. 该如何解决?】https://www.oschina.net/question/2417189_2264063?no_redirect=true

【kafka 消费能力不够?】https://bbs.csdn.net/topics/392512899

转载于:https://www.cnblogs.com/gospurs/p/10460653.html

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/正经夜光杯/article/detail/734978
推荐阅读
相关标签
  

闽ICP备14008679号