Kafka查看topic、consumer group状态命令_怎么查某个topic有哪些消费者组

作者：weixin_40725706 | 2024-04-10 01:53:25

踩

怎么查某个topic有哪些消费者组

最近工作中遇到需要使用kafka的场景，测试消费程序启动后，要莫名的过几十秒乃至几分钟才能成功获取到到topic的partition和offset，而后开始消费数据，于是学习了一下查看kafka broker里topic和consumer group状态的相关命令，这里记录一下。

命令参考自《Kafka: The Definitive Guide》 Chapter 9 Administrating Kafka

以下命令中使用的zookeeper配置地址为127.0.0.1:2181，bootstrap--server(即broker)地址为: 127.0.0.1:9292

1，查看kafka topic列表，使用--list参数


bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
__consumer_offsets
lx_test_topic
test

2，查看kafka特定topic的详情，使用--topic与--describe参数


bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic lx_test_topic --describe
Topic:lx_test_topic     PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: lx_test_topic    Partition: 0    Leader: 0       Replicas: 0     Isr: 0

列出了lx_test_topic的parition数量、replica因子以及每个partition的leader、replica信息

3，查看consumer group列表，使用--list参数

查看consumer group列表有新、旧两种命令，分别查看新版(信息保存在broker中)consumer列表和老版(信息保存在zookeeper中)consumer列表，因而需要区分指定bootstrap--server和zookeeper参数：


bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 127.0.0.1:9292 --list
lx_test


bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --list
console-consumer-86845
console-consumer-11967

4，查看特定consumer group 详情，使用--group与--describe参数

同样根据新/旧版本的consumer，分别指定bootstrap-server与zookeeper参数:


bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 127.0.0.1:9292 --group lx_test --describe
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
lx_test                        lx_test_topic             0          465             465             0               kafka-python-1.3.1_/127.0.0.1


bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --group console-consumer-11967 --describe
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
Could not fetch offset from zookeeper for group console-consumer-11967 partition [lx_test_topic,0] due to missing offset data in zookeeper.
console-consumer-11967         lx_test_topic             0          unknown         465             unknown         console-consumer-11967_aws-lx-1513787888172-d3a91f05-0

其中依次展示group名称、消费的topic名称、partition id、consumer group最后一次提交的offset、最后提交的生产消息offset、消费offset与生产offset之间的差值、当前消费topic-partition的group成员id(不一定包含hostname)

上面示例中console-consumer-11967是为了测试临时起的一个console consumer，缺少在zookeeper中保存的current_offset信息。

/home//kafka/kafka_2.11-1.0.0/bin> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server 192.168.80.11:9092 --group asset-kafka --describe
The --new-consumer option is deprecated and will be removed in a future major release.The new consumer is used by default if the --bootstrap-server option is provided.
Note: This will not show information about old Zookeeper-based consumers.

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test1 0 0 0 0 consumer-1-28c1186e-b790-4be8-a135-00a5fcd9448d /10.88.40.176 consumer-1
test 0 1 1 0 consumer-1-28c1186e-b790-4be8-a135-00a5fcd9448d /10.88.40.176 consumer-1
test4 0 0 0 0 consumer-1-28c1186e-b790-4be8-a135-00a5fcd9448d /10.88.40.176 consumer-1
test2 0 0 0 0 consumer-1-28c1186e-b790-4be8-a135-00a5fcd9448d /10.88.40.176 consumer-1
test3 0 0 0 0 consumer-1-28c1186e-b790-4be8-a135-00a5fcd9448d /10.88.40.176 consumer-1
test 3 0 0 0 consumer-1-87d63558-48bf-4dda-a894-33ec20a1ec86 /10.88.40.182 consumer-1
test 1 0 0 0 consumer-1-361af586-0f14-46df-99f4-a8f9d43ebca4 /10.88.40.180 consumer-1
test 4 0 0 0 consumer-1-bf0cbc55-951c-485b-8d10-a2f654686450 /10.88.40.190 consumer-1
test 2 0 0 0 consumer-1-845afd5c-952d-49ec-bacf-f4e9de458eb7 /10.88.40.147 consumer-1
/home//kafka/kafka_2.11-1.0.0/bin> ./kafka-consumer-groups.sh --zookeeper 192.168.80.11:27101 --group kafka.eagle.system.group --describe
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
__consumer_offsets 0 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 1 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 2 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 3 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 4 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 5 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 6 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 7 73 73 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 8 40 40 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 9 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 10 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 11 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 12 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 13 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 14 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 15 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 16 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 17 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 18 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 19 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 20 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 21 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 22 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 23 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 24 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 25 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 26 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 27 6513793 6513847 54 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 28 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 29 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 30 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 31 13071442 13071532 90 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 32 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 33 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 34 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 35 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 36 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 37 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 38 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 39 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 40 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd
__consumer_offsets 41 0 0 0 kafka.eagle.system.group_lvzhengtao-PC-1533105807135-de8ceffd

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/weixin_40725706/article/detail/395992