kafka重分配分区_kafka重新分配分区

1、确定需要执行重分区的topic，可以指定多个，建议每次重分区一个；（重分区之前检查下topic大小）

# vi topics-to-move.json
{
  "topics": [{
     "topic": "test-topic"
   }],
   "version": 1
}

2、生成分区方案

Current partition replica assignment 当前分区分布（用于回滚）

Proposed partition reassignment configuration  重分区之后的分区方案（分区后的效果）

注意broker-list指定重分配的节点，执行之前先确认每个节点的broker id

不一定需要执行改步骤，也可以手动分配

#bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":4,"replicas":[0,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":[1,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":[3,0],"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":[0,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":[0,2],"log_dirs":["any","any"]}]}
 
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":[1,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":[5,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":[3,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":[5,2],"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":[2,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":[4,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":[1,2],"log_dirs":["any","any"]}]}

手动分配分区文件如下所示：

{
	"version": 1,
	"partitions": [{
		"topic": "testTopic",
		"partition": 0,
		"replicas": [3, 0]
	}, {
		"topic": "testTopic",
		"partition": 1,
		"replicas": [0, 1]
	}, {
		"topic": "testTopic",
		"partition": 2,
		"replicas": [1, 2]
	}, {
		"topic": "testTopic",
		"partition": 3,
		"replicas": [2, 3]
	}, {
		"topic": "testTopic",
		"partition": 4,
		"replicas": [3, 1]
	}, {
		"topic": "testTopic",
		"partition": 5,
		"replicas": [0, 2]
	}, {
		"topic": "testTopic",
		"partition": 6,
		"replicas": [1, 3]
	}, {
		"topic": "testTopic",
		"partition": 7,
		"replicas": [2, 0]
	}, {
		"topic": "testTopic",
		"partition": 8,
		"replicas": [3, 2]
	}, {
		"topic": "testTopic",
		"partition": 9,
		"replicas": [0, 3]
	}]
}

3、将生成的分区方案保存至json文件中

#vi reassignment.json
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":[1,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":[5,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":[3,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":[5,2],"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":[2,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":[4,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":[1,2],"log_dirs":["any","any"]}]}

4、执行分区方案

 
# bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment.json --execute
Current partition replica assignment
 
{"version":1,"partitions":[{"topic":"test-topic","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":7,"replicas":[1,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":1,"replicas":[5,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":9,"replicas":[3,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":6,"replicas":[5,2],"log_dirs":["any","any"]},{"topic":"test-topic","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"test-topic","partition":8,"replicas":[2,4],"log_dirs":["any","any"]},{"topic":"test-topic","partition":0,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"test-topic","partition":5,"replicas":[4,1],"log_dirs":["any","any"]},{"topic":"test-topic","partition":2,"replicas":[1,2],"log_dirs":["any","any"]}]}
 
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

5、执行后登录Kafdrop检查分区状态

如果分区完成之后Preferred Replicas不是100%，需要手动重新为分区选举Leader

情况如下：

# cat leader.json 
{
    "version": 1,
    "partitions": [{
        "topic": "showItems",
        "partition": 0
    },{
        "topic": "showItems",
        "partition": 1
    },{
        "topic": "showItems",
        "partition": 2
    },{
        "topic": "showItems",
        "partition": 8
    },{
        "topic": "showItems",
        "partition": 9
    }]
}

重新选举

# bin/kafka-preferred-replica-election.sh --zookeeper localhost:2181 --path-to-json-file leader.json

执行后检查分区状态是否正常