kafka-重新分区_kafka增加分区,数据会重排吗

kafka 重新分区

为什么要重新分区？

kafka 不会因为 broker 的增加或者减少而重新对原有的分区进行扩展

重新分区命令

• –generate 生成分区重分配计划
• –execute 执行分区重分配计划
• –verify 验证分区充重配计划

准备

现集群：

$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED       STATUS        PORTS        NAMES
789ba7389459   zookeeper                            0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp               zoo
1d8d98bdc387   wurstmeister/kafka                   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                         kafka1
076eefa96bf5   wurstmeister/kafka                  0.0.0.0:9093->9092/tcp, :::9093->9092/tcp                          kafka2
c6ef4ee46fc1   sheepkiller/kafka-manager:latest      0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                        kafka-manager
855a13b1317a   wurstmeister/kafka                   0.0.0.0:9094->9092/tcp, :::9094->9092/tcp                         kafka3

bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --create --replication-factor 2 --partitions 4  --topic my-topic
Created topic my-topic.
bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
my-topic

bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic  --describe
Topic: my-topic TopicId: -qrwf9nwTUmd4JjrOr1ftQ PartitionCount: 4       ReplicationFactor: 2    Configs: segment.bytes=1073741824
        Topic: my-topic Partition: 0    Leader: 3       Replicas: 3,1   Isr: 3,1
        Topic: my-topic Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: my-topic Partition: 2    Leader: 2       Replicas: 2,3   Isr: 2,3
        Topic: my-topic Partition: 3    Leader: 3       Replicas: 3,2   Isr: 3,2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

my-topic 4个分区，两个副本

场景： 模拟 broker3 掉线， 对 my-topic 进行重新 分区

生成分区重分配计划

重新分配 json:

bash-5.1# cat resign-plan.json 
{
        "topics": [{
                "topic": "my-topic"
        }],
        "version": 1
}
1
2
3
4
5
6
7

生成分配计划

bash-5.1# kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --topics-to-move-json-file resign-plan.json --broker-list "1,2" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"my-topic","partition":1,"replicas":[3,1],"log_dirs":["any","any"]},{"topic":"my-topic","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"my-topic","partition":3,"replicas":[2,1],"log_dirs":["any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"my-topic","partition":3,"replicas":[1,2],"log_dirs":["any","any"]}]}
1
2
3
4
5
6

• 第一个json 是当前的分区副本分配 情况，可以 做备份回滚
• 第二个json 是 生成 可行性分区分配方案

执行分区重分配计划

将 第二个 json 保存为 resign.json

bash-5.1# kafka-reassign-partitions.sh --bootstrap-server 192.168.xx.x:9092 --reassignment-json-file resign.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"my-topic","partition":1,"replicas":[3,1],"log_dirs":["any","any"]},{"topic":"my-topic","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"my-topic","partition":3,"replicas":[2,1],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for my-topic-0,my-topic-1,my-topic-2,my-topic-3
1
2
3
4
5
6
7

验证分区重分配计划

–reassignment-json-file 指定JSON格式配置文件，第一步生成的重分配计划

bash-5.1# kafka-reassign-partitions.sh --bootstrap-server 192.168.xx.x:9092 --reassignment-json-file resign.json --verify
Status of partition reassignment:
Reassignment of partition my-topic-0 is complete.
Reassignment of partition my-topic-1 is complete.
Reassignment of partition my-topic-2 is complete.
Reassignment of partition my-topic-3 is complete.

Clearing broker-level throttles on brokers 1,2,3
Clearing topic-level throttles on topic my-topic
1
2
3
4
5
6
7
8
9

查看分区:

bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic  --describe
Topic: my-topic TopicId: iDyTLdtERzerRP1nYtJOCg PartitionCount: 4       ReplicationFactor: 2    Configs: segment.bytes=1073741824
        Topic: my-topic Partition: 0    Leader: 2       Replicas: 2,1   Isr: 2,1
        Topic: my-topic Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: my-topic Partition: 2    Leader: 1       Replicas: 2,1   Isr: 1,2
        Topic: my-topic Partition: 3    Leader: 2       Replicas: 1,2   Isr: 2,1
1
2
3
4
5
6

**可以发现，分区都被重新分配到 broker 1，2 节点上，可以安全的移除broker3节点 **