知其然,不知其所以然的写代码,不是目的。
今天etcd集群的一个节点挂了,简单写下处理过程。raft算法春节期间必须仔细读一下。
从集群中剔除有问题的节点
//在正常的节点执行如下命令,获取到不正常节点的ID,hostB节点坏掉
# /usr/local/etcd/etcdctl member list
4121a0b16ac4c150: name=hostA peerURLs=http://10.33.1.10:2380 clientURLs=http://10.33.1.10:2379
6b7887955883c31a: name=hostB peerURLs=http://10.33.1.11:2380 clientURLs=http://10.33.1.11:2379
d8dd30b565cff95c: name=hostC peerURLs=http://10.33.1.12:2380 clientURLs=http://10.33.1.12:2379
# /usr/local/etcd/etcdctl cluster-health
cluster is healthy
member 4121a0b16ac4c150 is healthy
member 6b7887955883c31a is unhealthy
member d8dd30b565cff95c is healthy
//删除掉坏的节点
# /usr/local/etcd/etcdctl member remove 6b7887955883c31a
修复坏的节点
启动的命令
-initial-cluster-state 由"new" 改成"existing"
在启动新节点之前,必须把新节点接入集群,然后在启动新节点,,否则会出现
"etcd: error validating peerURLs hostA=http://10.33.1.10:2380,hostC=http://10.33.1.12:2380: member count is unequal"
的报错
# /usr/local/etcd/etcdctl member add hostD http://10.33.1.15:2380
Added member named hostD with ID 2519d1a6a8cb83da to cluster
新节点启动命令:
/usr/local/etcd/etcd -name hostD -initial-advertise-peer-urls http://10.33.1.15:2380 -listen-peer-urls http://10.33.1.15:2380 -listen-client-urls http://10.33.1.15:2379,http://127.0.0.1:2379 -advertise-client-urls http://10.33.1.15:2379 -initial-cluster-token etcd-cluster-aa -initial-cluster hostA=http://10.33.1.10:2380,hostC=http://10.33.1.12:2380,hostD=http://10.33.1.15:2380 -initial-cluster-state existing &
查看集群状态:
# etcdctl cluster-health
cluster is healthy
member 2519d1a6a8cb83da is healthy
member 4121a0b16ac4c150 is healthy
member d8dd30b565cff95c is healthy