一、 分区副本重分配
1 生成推荐配置脚本
关键参数--generate
在进行分区副本重分配之前,最好是用下面方式获取一个合理的分配文件;
编写move-json-file.json文件; 这个文件就是告知想对哪些topic进行重新分配的计算。
$ cat move-json-file.json
{
"topics": [
{"topic": "topic_name"}
],
"version": 1
}
然后执行下面的脚本,--broker-list "0,1,2" 这个参数是你想要分配的Brokers。
$ sh bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181 --topics-to-move-json-file config/move-json-file.json --broker-list "0,1,2" --generate
执行完毕之后会打印：
Current partition replica assignment
{"version":1,"partitions":[{"topic":"topic_name","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":3,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":8,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":6,"replicas":[0],"log_dirs":["any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"topic_name","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"topic_name","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"topic_name","partition":6,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":3,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"topic_name","partition":8,"replicas":[2],"log_dirs":["any"]}]}
Current partition replica assignment
{
"version": 1,
"partitions": [{
"topic": "topic_name",
"partition": 4,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 1,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 2,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 0,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 7,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 3,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 8,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 5,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 6,
"replicas": [0],
"log_dirs": ["any"]
}]
}
Proposed partition reassignment configuration
{
"version": 1,
"partitions": [{
"topic": "topic_name",
"partition": 2,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 5,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 7,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 4,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 1,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 6,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 3,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 0,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 8,
"replicas": [2],
"log_dirs": ["any"]
}]
}
需求注意的是，此时分区移动尚未开始，它只是告诉你当前的分配和建议。保存当前分配，以防你想要回滚它。
2. 执行json文件
关键参数--execute将上面得到期望的重新分配方式文件保存在一个json文件里面 reassignment-json-file.json
$ cat reassignment-json-file.json
{
"version": 1,
"partitions": [{
"topic": "topic_name",
"partition": 2,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 5,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 7,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 4,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 1,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 6,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 3,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 0,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 8,
"replicas": [2],
"log_dirs": ["any"]
}]
}
$ sh bin/kafka-reassign-partitions.sh --zookeeper xx.xx.xx.xx:2181 --reassignment-json-file config/reassignment-json-file.json --execute
二、 副本扩缩
kafka并没有提供一个专门的脚本来支持副本的扩缩, 不像kafka-topic.sh脚本一样,是可以扩分区的;
想要对副本进行扩缩,只能是曲线救国，利用kafka-reassign-partitions.sh来重新分配副本。
副本扩容
假设我们当前的情况是 3分区1副本,为了提供可用性,我想把副本数升到2;
计算副本分配方式
我们用 --generate 获取一下当前的分配情况,得到如下json
Current partition replica assignment
{
"version": 1,
"partitions": [{
"topic": "topic_name",
"partition": 4,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 1,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 2,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 0,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 7,
"replicas": [1],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 3,
"replicas": [0],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 8,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 5,
"replicas": [2],
"log_dirs": ["any"]
}, {
"topic": "topic_name",
"partition": 6,
"replicas": [0],
"log_dirs": ["any"]
}]
}
三、参考
数据迁移、分区副本重分配、跨路径迁移、副本扩缩容
https://developer.aliyun.com/article/785752
https://www.szzdzhp.com/kafka/op/op-partition-reasignment.html
kafka最小成本的扩缩容副本设计方案
https://blog.csdn.net/u010634066/article/details/120931626
你不知道的kafka配置broker.id
https://cloud.tencent.com/developer/news/378568
json 格式化校验
https://www.bejson.com
kafka修改分区、副本数、副本迁移
https://sukbeta.github.io/kafka-Modify-Partitions-and-ReplicationFactor
kafka扩容副本数
https://www.cnblogs.com/mysql-hang/articles/14327103.html
