devbox
从前慢现在也慢
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

kafka集群broker节点扩容方案_kafka扩容节点

作者：从前慢现在也慢 | 2024-05-29 16:43:13

赞

踩

kafka扩容节点
在用kafka集群有3个节点，即host1, host2, host3，现需要将broker节点扩容至6个，以提供更高的数据处理能力。
一、上架物理服务器用于提供更多的资源
新扩容3个broker节点，host4,host5,host6

二、在三台新增节点上部署kafka应用程序
这些不是本文的重点，略过。

三、重新分布原有的topic分区

1、查看集群中当前所有可用的topic
./kafka-topics.sh --list --zookeeper ip:port
  1. lcf-201612201649
  2. test-for-sys-monitor
2、查看特定topic的详细信息 
  1. ./kafka-topics.sh --describe --zookeeper 192.168.1.92:2181 --topic lcf-201612201649
  2. Topic:lcf-201612201649    PartitionCount:24    ReplicationFactor:3    Configs:
  3.     Topic: lcf-201612201649    Partition: 0    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2
  4.     Topic: lcf-201612201649    Partition: 1    Leader: 2    Replicas: 2,1,3    Isr: 3,1,2
  5.     Topic: lcf-201612201649    Partition: 2    Leader: 3    Replicas: 3,2,1    Isr: 3,1,2
  6.     Topic: lcf-201612201649    Partition: 3    Leader: 1    Replicas: 1,2,3    Isr: 1,3,2
  7.     Topic: lcf-201612201649    Partition: 4    Leader: 2    Replicas: 2,3,1    Isr: 3,1,2
  8.     Topic: lcf-201612201649    Partition: 5    Leader: 3    Replicas: 3,1,2    Isr: 3,1,2
  9.     Topic: lcf-201612201649    Partition: 6    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2
  10.     Topic: lcf-201612201649    Partition: 7    Leader: 2    Replicas: 2,1,3    Isr: 1,3,2
  11.     Topic: lcf-201612201649    Partition: 8    Leader: 3    Replicas: 3,2,1    Isr: 3,1,2
  12.     Topic: lcf-201612201649    Partition: 9    Leader: 1    Replicas: 1,2,3    Isr: 1,3,2
  13.     Topic: lcf-201612201649    Partition: 10    Leader: 2    Replicas: 2,3,1    Isr: 3,1,2
  14.     Topic: lcf-201612201649    Partition: 11    Leader: 3    Replicas: 3,1,2    Isr: 3,1,2
  15.     Topic: lcf-201612201649    Partition: 12    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2
  16.     Topic: lcf-201612201649    Partition: 13    Leader: 2    Replicas: 2,1,3    Isr: 3,1,2
  17.     Topic: lcf-201612201649    Partition: 14    Leader: 3    Replicas: 3,2,1    Isr: 3,1,2
  18.     Topic: lcf-201612201649    Partition: 15    Leader: 1    Replicas: 1,2,3    Isr: 1,3,2
  19.     Topic: lcf-201612201649    Partition: 16    Leader: 2    Replicas: 2,3,1    Isr: 3,1,2
  20.     Topic: lcf-201612201649    Partition: 17    Leader: 3    Replicas: 3,1,2    Isr: 3,1,2
  21.     Topic: lcf-201612201649    Partition: 18    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2
  22.     Topic: lcf-201612201649    Partition: 19    Leader: 2    Replicas: 2,1,3    Isr: 1,3,2
  23.     Topic: lcf-201612201649    Partition: 20    Leader: 3    Replicas: 3,2,1    Isr: 3,1,2
  24.     Topic: lcf-201612201649    Partition: 21    Leader: 1    Replicas: 1,2,3    Isr: 1,3,2
  25.     Topic: lcf-201612201649    Partition: 22    Leader: 2    Replicas: 2,3,1    Isr: 3,1,2
  26.     Topic: lcf-201612201649    Partition: 23    Leader: 3    Replicas: 3,1,2    Isr: 3,1,2

3、kafka集群分区日志迁移
我们的目标是把原来分布在3个节点上的topic（24partitions, 3replicas），将全部partitions重新分布到全部的6个节点上去。

(1) 先制作topics-to-move.json文件 
  1. cat << EOF > topic-to-move.json
  2. {"topics": [{"topic": "lcf-201612201649"}],
  3. "version":1
  4. }
  5. EOF

(2)使用-generate生成迁移计划 
  1. [testuser@c4 bin]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --topics-to-move-json-file  ./plans/topic-to-move.json  --broker-list "1,2,3,4,5,6" --generate
  2. Current partition replica assignment
  3. {"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":19,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":18,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":0,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":10,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":5,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":12,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":17,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":9,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":3,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":2,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":4,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":14,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":22,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":21,"replicas":[1,2,3]}]}
  4. Proposed partition reassignment configuration
  5. {"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[4,1,2]},{"topic":"lcf-201612201649","partition":15,"replicas":[6,5,1]},{"topic":"lcf-201612201649","partition":8,"replicas":[5,3,4]},{"topic":"lcf-201612201649","partition":19,"replicas":[4,5,6]},{"topic":"lcf-201612201649","partition":13,"replicas":[4,3,5]},{"topic":"lcf-201612201649","partition":18,"replicas":[3,4,5]},{"topic":"lcf-201612201649","partition":0,"replicas":[3,6,1]},{"topic":"lcf-201612201649","partition":10,"replicas":[1,5,6]},{"topic":"lcf-201612201649","partition":5,"replicas":[2,5,6]},{"topic":"lcf-201612201649","partition":12,"replicas":[3,2,4]},{"topic":"lcf-201612201649","partition":9,"replicas":[6,4,5]},{"topic":"lcf-201612201649","partition":17,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[4,2,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[5,6,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[2,3,4]},{"topic":"lcf-201612201649","partition":3,"replicas":[6,3,4]},{"topic":"lcf-201612201649","partition":4,"replicas":[1,4,5]},{"topic":"lcf-201612201649","partition":2,"replicas":[5,2,3]},{"topic":"lcf-201612201649","partition":11,"replicas":[2,6,1]},{"topic":"lcf-201612201649","partition":6,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":22,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":14,"replicas":[5,4,6]},{"topic":"lcf-201612201649","partition":16,"replicas":[1,6,2]},{"topic":"lcf-201612201649","partition":21,"replicas":[6,1,2]}]}
  6. [testuser@c4 bin]$
注：生成一个向broker 1,2,3,4,5,6迁移指定topic的迁移计划。输出内容中包括当前的分布配置和即将改变后的分布配置。
将以上命令的输出内容保存为json文件。其中当前分布配置备份为backup.json，改变后的分布配置保存为expand-cluster-reassignment.json

(3)使用-execute执行迁移计划 
  1. [testuser@c4 kafka]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --reassignment-json-file ./plans/reassignment-lcf-201612201649.json --execute
  2. Current partition replica assignment
  3. {"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":19,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":18,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":0,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":10,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":5,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":12,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":17,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":9,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":3,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":2,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":4,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":14,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":22,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":21,"replicas":[1,2,3]}]}
  4.  
  5. Save this to use as the --reassignment-json-file option during rollback
  6. Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"lcf-201612201649","partition":12,"replicas":[3,2,4]},{"topic":"lcf-201612201649","partition":17,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[5,3,4]},{"topic":"lcf-201612201649","partition":7,"replicas":[4,2,3]},{"topic":"lcf-201612201649","partition":23,"replicas":[2,3,4]},{"topic":"lcf-201612201649","partition":4,"replicas":[1,4,5]},{"topic":"lcf-201612201649","partition":19,"replicas":[4,5,6]},{"topic":"lcf-201612201649","partition":2,"replicas":[5,2,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[5,6,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[2,6,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[1,6,2]},{"topic":"lcf-201612201649","partition":5,"replicas":[2,5,6]},{"topic":"lcf-201612201649","partition":14,"replicas":[5,4,6]},{"topic":"lcf-201612201649","partition":10,"replicas":[1,5,6]},{"topic":"lcf-201612201649","partition":18,"replicas":[3,4,5]},{"topic":"lcf-201612201649","partition":22,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[6,5,1]},{"topic":"lcf-201612201649","partition":1,"replicas":[4,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":21,"replicas":[6,1,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[4,3,5]},{"topic":"lcf-201612201649","partition":0,"replicas":[3,6,1]},{"topic":"lcf-201612201649","partition":9,"replicas":[6,4,5]},{"topic":"lcf-201612201649","partition":3,"replicas":[6,3,4]}]}

(4)使用--verify进行迁移结果的验证 
[testuser@c4 kafka]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --reassignment-json-file ./plans/reassignment-lcf-201612201649.json --verify

查看重分布结果： 
./bin/kafka-topics.sh --describe --zookeeper 192.168.1.92:2181 --topic lcf-201612201649

当以上迁移过程导致kafka的leader分布，不符preferred replica分布建议，则可以手动进行再平衡维护。
注：进行分区迁移时，最好先保留一个分区在原来的磁盘，这样不会影响正常的消费和生产。部分迁移则支持正常消费和生产。

(5)关于kafka集群leader平衡机制的维护 
./kafka-preferred-replica-election.sh --zookeeper ip:port
或者在配置文件中将参数设置为： 
auto.leader.rebalance.enable=true


声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/从前慢现在也慢/article/detail/643181
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号