如何给 Kafka 新增分区_kafka 增加分区

数据量猛增的时候，需要给 kafka 的 topic 新增分区，增大处理的数据量，可以通过以下步骤

1、修改 topic 的分区

kafka-topics --zookeeper hadoop004:2181 --alter --topic flink-test-04 --partitions 3

2、迁移数据

生成迁移计划，手动新建一个 json 文件

{
"topics": [
{"topic": "flink-test-03"}
],
"version": 1
}
1
2
3
4
5
6

生成迁移计划
kafka-reassign-partitions --zookeeper hadoop004:2181 --topics-to-move-json-file topic.json --broker-list “120,121,122” --generate

Current partition replica assignment:

{"version":1,"partitions":[{"topic":"flink-test-02","partition":5,"replicas":[120]},{"topic":"flink-test-02","partition":0,"replicas":[121]},{"topic":"flink-test-02","partition":2,"replicas":[120]},{"topic":"flink-test-02","partition":1,"replicas":[122]},{"topic":"flink-test-02","partition":4,"replicas":[122]},{"topic":"flink-test-02","partition":3,"replicas":[121]}]}
1

新建一个文件reassignment.json，保存上边这些信息

3、迁移

kafka-reassign-partitions --zookeeper hadoop004:2181 --reassignment-json-file reassignment.json --execute
1

4、验证

kafka-reassign-partitions --zookeeper hadoop004:2181 --reassignment-json-file reassignment.json --verify
1