【云原生】第五章 k8s基础组件-部署Etcd集群

Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。

1、节点地址

节点名称 节点地址 hostsname
Etcd-1 172.21.209.32 k8s-master01
Etcd-2 172.21.209.33 k8s-master02
Etcd-3 172.21.209.34 k8s-master03

2、etcd下载

最新版本：https://github.com/etcd-io/etcd/releases/download/

wget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz

3、分发二进制包到其他的etcd节点

for  i in {32..34};do scp -P 22022 ./etcd-v3.5.4-linux-amd64.tar.gz 172.21.209.$i:/data/; done

说明：-P表示端口。


#或者将刚刚解压的二进制文件拷贝到其它服务器上
for i in k8s-master02 k8s-master03;do
    scp -r -P 22022 /usr/local/bin/kube* root@$i:/usr/local/bin/
    scp  -r -P 22022 /usr/local/bin/{etcd,etcdctl}   root@$i:/usr/local/bin/
done

4、解压二进制包并完成安装

tar -xf etcd-v3.5.4-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.4-linux-amd64/etcd{,ctl}

查看版本，安装完成。
root@k8s-master02:/data# etcd --version
etcd Version: 3.5.4
Git SHA: 08407ff76
Go Version: go1.16.15
Go OS/Arch: linux/amd64
root@k8s-master02:/data#

5、分发证书到其他节点上。

1、创建证书存放目录并，将做好的证书拷贝到其他节点上
批量执行 #master的所有节点执行
for i in k8s-master02 k8s-master03;do
    ssh -p 22022  $i "mkdir  /etc/kubernetes/pki/ -p"
    scp -r -P 22022 /etc/kubernetes/pki $i:/etc/kubernetes/
done

查看节点02上的证书
root@k8s-master02:/etc/kubernetes/pki# ll
total 112
drwxr-xr-x 2 root root 4096 Dec 13 16:13 ./
drwxr-xr-x 3 root root 4096 Dec 13 16:13 ../
-rw-r--r-- 1 root root 1025 Dec 13 16:13 admin.csr
-rw------- 1 root root 1679 Dec 13 16:13 admin-key.pem
-rw-r--r-- 1 root root 1444 Dec 13 16:13 admin.pem
-rw-r--r-- 1 root root 1029 Dec 13 16:13 apiserver.csr
-rw------- 1 root root 1679 Dec 13 16:13 apiserver-key.pem
-rw-r--r-- 1 root root 1996 Dec 13 16:13 apiserver.pem
-rw-r--r-- 1 root root 1025 Dec 13 16:13 ca.csr
-rw------- 1 root root 1675 Dec 13 16:13 ca-key.pem
-rw-r--r-- 1 root root 1411 Dec 13 16:13 ca.pem
-rw-r--r-- 1 root root 1082 Dec 13 16:13 controller-manager.csr
-rw------- 1 root root 1679 Dec 13 16:13 controller-manager-key.pem
-rw-r--r-- 1 root root 1501 Dec 13 16:13 controller-manager.pem
-rw-r--r-- 1 root root  891 Dec 13 16:13 front-proxy-ca.csr
-rw------- 1 root root 1675 Dec 13 16:13 front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1143 Dec 13 16:13 front-proxy-ca.pem
-rw-r--r-- 1 root root  903 Dec 13 16:13 front-proxy-client.csr
-rw------- 1 root root 1679 Dec 13 16:13 front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Dec 13 16:13 front-proxy-client.pem
-rw-r--r-- 1 root root 1045 Dec 13 16:13 kube-proxy.csr
-rw------- 1 root root 1675 Dec 13 16:13 kube-proxy-key.pem
-rw-r--r-- 1 root root 1464 Dec 13 16:13 kube-proxy.pem
-rw------- 1 root root 1679 Dec 13 16:13 sa.key
-rw-r--r-- 1 root root  451 Dec 13 16:13 sa.pub
-rw-r--r-- 1 root root 1058 Dec 13 16:13 scheduler.csr
-rw------- 1 root root 1679 Dec 13 16:13 scheduler-key.pem
-rw-r--r-- 1 root root 1476 Dec 13 16:13 scheduler.pem
root@k8s-master02:/etc/kubernetes/pki#

6、配置ETCD

k8s-master01配置文件，请根据需求修改


# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.21.209.32:2380'
listen-client-urls: 'https://172.21.209.32:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.21.209.32:2380'
advertise-client-urls: 'https://172.21.209.32:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://172.21.209.32:2380,k8s-master02=https://172.21.209.33:2380,k8s-master03=https://172.21.209.34:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

k8s-master02配置文件，请根据需求修改


# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.21.209.33:2380'
listen-client-urls: 'https://172.21.209.33:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.21.209.33:2380'
advertise-client-urls: 'https://172.21.209.33:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://172.21.209.32:2380,k8s-master02=https://172.21.209.33:2380,k8s-master03=https://172.21.209.34:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

k8s-master03配置文件，请根据需求修改


# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.21.209.34:2380'
listen-client-urls: 'https://172.21.209.34:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.21.209.34:2380'
advertise-client-urls: 'https://172.21.209.34:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://172.21.209.32:2380,k8s-master02=https://172.21.209.33:2380,k8s-master03=https://172.21.209.34:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/etcd/etcd.pem'
  key-file: '/etc/kubernetes/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

7、创建etcd启动服务（需要在所有master节点操作）

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

8、启动服务

systemctl daemon-reload
systemctl enable --now etcd

9、查看etcd状态


# 如果要用IPv6那么把IPv4地址修改为IPv6即可
root@k8s-master02:~# export ETCDCTL_API=3
root@k8s-master02:~# etcdctl --endpoints="k8s-master01:2379,k8s-master02:2379,k8s-master03:2379" --cacert=/etc/kubernetes/etcd/etcd-ca.pem --cert=/etc/kubernetes/etcd/etcd.pem --key=/etc/kubernetes/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| k8s-master01:2379 | 8d64b715c092a95e |   3.5.4 |   20 kB |     false |      false |         3 |         18 |                 18 |        |
| k8s-master02:2379 | 360eef617d4fed2c |   3.5.4 |   20 kB |      true |      false |         3 |         18 |                 18 |        |
| k8s-master03:2379 | 344830ef5ebe5c27 |   3.5.4 |   20 kB |     false |      false |         3 |         18 |                 18 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
root@k8s-master02:~#

10、etcd的常用操作

#查看 etcd 集群成员列表
root@k8s-master02:~#  etcdctl member list
344830ef5ebe5c27, started, k8s-master03, https://172.21.209.34:2380, https://172.21.209.34:2379, false
360eef617d4fed2c, started, k8s-master02, https://172.21.209.33:2380, https://172.21.209.33:2379, false
8d64b715c092a95e, started, k8s-master01, https://172.21.209.32:2380, https://172.21.209.32:2379, false
root@k8s-master02:~# 



# 删除 etcd 集群成员 k8s-master-2-11
 etcdctl member remove 344830ef5ebe5c27

问题：

问题：查看etcd的状态时，只有一个在线，并且其他的节点的服务也是正常的。无法获取集群状态，但是通过地址就可以，经过排除发现证书有问题，只对k8s-master01做了授权，没有对k8s-master02和k8s-master03授权。通过IP地址可以正常获取状态。

root@k8s-master01:~# export ETCDCTL_API=3
root@k8s-master01:~# etcdctl --endpoints="k8s-master01:2379,k8s-master02:2379,k8s-master03:2379" --cacert=/etc/kubernetes/etcd/etcd-ca.pem --cert=/etc/kubernetes/etcd/etcd.pem --key=/etc/kubernetes/etcd/etcd-key.pem  endpoint status --write-out=table

{"level":"warn","ts":"2022-12-13T19:57:47.382+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000444380/k8s-master01:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for k8s-master01, k8s-master01, k8s-master01, not k8s-master02\""}
Failed to get the status of endpoint k8s-master02:2379 (context deadline exceeded)
{"level":"warn","ts":"2022-12-13T19:57:52.382+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000444380/k8s-master01:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for k8s-master01, k8s-master01, k8s-master01, not k8s-master03\""}
Failed to get the status of endpoint k8s-master03:2379 (context deadline exceeded)
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| k8s-master01:2379 | 8d64b715c092a95e |   3.5.4 |   20 kB |     false |      false |         2 |         14 |                 14 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
root@k8s-master01:~# 
root@k8s-master01:~# 


通过主机名无法获取，但是通过ip可以获取。
etcdctl --endpoints="k8s-master02:2379" --cacert=/etc/kubernetes/etcd/etcd-ca.pem --cert=/etc/kubernetes/etcd/etcd.pem --key=/etc/kubernetes/etcd/etcd-key.pem  endpoint status --write-out=table



root@k8s-master03:/etc/etcd# etcdctl --endpoints="172.21.209.34:2379" --cacert=/etc/kubernetes/etcd/etcd-ca.pem --cert=/etc/kubernetes/etcd/etcd.pem --key=/etc/kubernetes/etcd/etcd-key.pem  endpoint status --write-out=table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 172.21.209.34:2379 | 344830ef5ebe5c27 |   3.5.4 |   20 kB |      true |      false |         2 |         14 |                 14 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
root@k8s-master03:/etc/etcd# 
root@k8s-master03:/etc/etcd# 
root@k8s-master03:/etc/etcd# 
root@k8s-master03:/etc/etcd# etcdctl --endpoints="172.21.209.33:2379" --cacert=/etc/kubernetes/etcd/etcd-ca.pem --cert=/etc/kubernetes/etcd/etcd.pem --key=/etc/kubernetes/etcd/etcd-key.pem  endpoint status --write-out=table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 172.21.209.33:2379 | 360eef617d4fed2c |   3.5.4 |   20 kB |     false |      false |         2 |         14 |                 14 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
root@k8s-master03:/etc/etcd#

至此：ETCD集群部署完成。

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 203,547评论 6赞 477
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 85,399评论 2赞 381
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 150,428评论 0赞 337
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 54,599评论 1赞 274
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 63,612评论 5赞 365
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 48,577评论 1赞 281
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 37,941评论 3赞 395
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 36,603评论 0赞 258
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 40,852评论 1赞 297
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 35,605评论 2赞 321
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 37,693评论 1赞 329
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 33,375评论 4赞 318
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 38,955评论 3赞 307
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 29,936评论 0赞 19
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,172评论 1赞 259
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 43,970评论 2赞 349
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 42,414评论 2赞 342

推荐阅读更多精彩内容