建议使用阿里云自身的容器服务吧，毕竟自建k8s集群太折腾
1、资源架构
前期使用 2 master + 3 worker配置，后期再扩展worker节点
| 资源、主机名 | IP | 配置 | 服务 |
|---|---|---|---|
| SLB1-ALB | 172.18.20.55 | 内网基础版 | master VIP |
| SLB2 | 外网IP，应用服务 | ingress | |
| k8s-master1 | 172.18.20.44 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-master2 | 172.18.20.45 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-master3 | 172.18.20.46 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-worker1 | 172.18.20.47 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
| k8s-worker2 | 172.18.20.48 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
| k8s-worker3 | 172.18.20.49 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
架构图
堆叠（Stacked） etcd 拓扑
堆叠的 etcd 拓扑
外部 etcd 拓扑
外部 etcd 拓扑
2、基础环境
2.1、软件版本
由于kubernetes 1.20 版本开始已弃用docker，推荐使用containerd
| 软件 | 版本 |
|---|---|
| os | CentOS 7.9 |
| containerd | 1.4.3 |
| Kubernetes | 1.21 |
| etcd | 3.4 |
2.2、系统配置
没有特别说明的，所有主机需执行
- 修改hostname
hostname k8s-master1
- 关闭防火墙
# systemctl stop firewalld
# systemctl disable firewalld
- 关闭selinux
# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
# setenforce 0 # 临时
- 关闭 swap
# swapoff -a # 临时
# sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
- 添加 hosts
# cat >> /etc/hosts << EOF
172.18.20.44 k8s-master1
172.18.20.45 k8s-master2
172.18.20.46 k8s-master3
...
EOF
- 配置节点主机免密访问(master)
# ssh-keygen
# ssh-copy-id root@k8s-*
- 将桥接的IPv4流量传递到iptables的链
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl --system
-
调整系统内核参数
# cat > /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv6.conf.all.disable_ipv6=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 fs.file-max=2000000 fs.nr_open=2000000 fs.inotify.max_user_instances=512 fs.inotify.max_user_watches=1280000 net.netfilter.nf_conntrack_max=524288 EOF # modprobe br_netfilter && sysctl -p /etc/sysctl.d/kubernetes.conf -
错误提示
/proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录记得运行
modprobe br_netfilter -
加载ipvs 模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules sh /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_
3、etcd 集群部署
k8s集群使用堆叠 ETCD 时，可不单独创建，本人是为了延长 etcd 证书才单独部署集群，实际架构也是堆叠式
3.1 使用 etcdadm 工具
下载
# wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.3/etcdadm-linux-amd64
# mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
# chmod +x /usr/local/bin/etcdadm
选择一个节点初始化etcd
etcdadm init --install-dir="/opt/etcd/" --name=etcd-1
参数解析
- --install-dir 安装目录
- --name 节点名称
- server-cert-extra-sans 负载均衡地址（单独部署使用）
拷贝证书到其他节点
scp /etc/etcd/pki/ca.* root@master2:/etc/etcd/pki/
scp /etc/etcd/pki/ca.* root@master3:/etc/etcd/pki/
加入集群
etcdadm join https://172.18.20.44:2379 --install-dir="/opt/etcd/" --name=etcd-2
安装完成后查看集群
修改环境变量
source /etc/etcd/etcdctl.env
查看
# /opt/bin/etcdctl member list
18ed9897779ce358, started, k8s-master1, https://172.18.20.50:2380, https://172.18.20.50:2379, false
3e04e89693dd5c26, started, k8s-master2, https://172.18.20.51:2380, https://172.18.20.51:2379, false
68562c67a29d6f06, started, k8s-master3, https://172.18.20.49:2380, https://172.18.20.49:2379, false
建议直接使用 etcdctl.sh
注意：etcdadm 创建的证书期限为一年。需要一年更换一次，这对生产环境来说很不友好。目前有两个解决方案
1、二进制部署 etcd，使用openssl 生成证书
-
2、修改etcdadm源码
下载源码
git clone https://github.com/kubernetes-sigs/etcdadm.git
修改 certs/pkiutil/pki_helpers.go 文件
certificateValidity =time.Hour * 24 * 365 * 10 # 10年期限
const (
// PrivateKeyBlockType is a possible value for pem.Block.Type.
PrivateKeyBlockType = "PRIVATE KEY"
// PublicKeyBlockType is a possible value for pem.Block.Type.
PublicKeyBlockType = "PUBLIC KEY"
// CertificateBlockType is a possible value for pem.Block.Type.
CertificateBlockType = "CERTIFICATE"
// RSAPrivateKeyBlockType is a possible value for pem.Block.Type.
RSAPrivateKeyBlockType = "RSA PRIVATE KEY"
rsaKeySize = 2048
certificateValidity = time.Hour * 24 * 365 * 10
)
编译（需要golang 1.15 以上版本）
make
编译完成后使用 etcdadm 重新生成集群
3.2、二进制部署
太麻烦了，网上找资料吧。ectdadm非常nice
4、安装 docker 和 containerd (所有节点)
1.20 版本模式使用 containerd ，可不用安装docker
添加 yum 源
# yum install -y yum-utils
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安装
yum install -y containerd.io
生成默认配置
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
添加私有镜像仓库
containerd 修改 config.toml 配置
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
# 阿里云私有镜像仓库
[plugins."io.containerd.grpc.v1.cri".cri.registry.mirrors."registry-vpc.cn-shenzhen.aliyuncs.com"]
endpoint = ["http://registry-vpc.cn-shenzhen.aliyuncs.com"]
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.auths."registry-vpc.cn-shenzhen.aliyuncs.com"]
username = "username"
password = "password
启动
systemctl restart containerd
systemctl enable containerd
5、使用kubeadm部署高可用集群
5.1、安装kubeadm
yum源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
安装
yum install -y kubeadm kubelet kubectl
无法使用kubeadm启动kubelet时，需要修改 kubelet 使用 container
# vim /usr/lib/systemd/system/kubelet.services.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
5.2、初始化集群
kubeadm默认会创建一个堆叠式的etcd集群，并不需要另外部署
kubeadm init --control-plane-endpoint "k8s-master-slb:6443" --upload-certs --node-name "k8s-master1"
参数解析
- --control-plane-endpoint apiserver集群地址
- --upload-certs 证书
由于国内无法访问google的镜像地址，这里需要使用国内的地址来下载镜像，下载完成后需要更新 tag 为 k8s.gcr.io
5.3、使用外部 etcd 初始化集群
创建 kubeadm 初始化配置文件
# 生成 kubeadm 默认初始化模板
kubeadm config print init-defaults > kubeadm-config.yaml
kubeadm-config 使用说明 https://blog.51cto.com/foxhound/2517491?source=dra
修改
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
localAPIEndpoint:
advertiseAddress: 192.168.200.125
bindPort: 6443
nodeRegistration:
kubeletExtraArgs:
runtime-cgroups: /system.slice/containerd.service
kubelet-cgroups: /systemd/system.slice
container-runtime: remote
#container-runtime-endpoint: unix:///var/run/docker.sock
container-runtime-endpoint: unix:///run/containerd/containerd.sock
cgroup-driver: systemd
#criSocket: /var/run/docker.sock
criSocket: /run/containerd/containerd.sock
name: k8s-master1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.4
networking:
dnsDomain: cluster.local
podSubnet: 10.10.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
# 负载均衡地址
controlPlaneEndpoint: "k8s-master-slb:7443"
apiServer:
timeoutForControlPlane: 10m0s
certificatesDir: /etc/kubernetes/pki
clusterName: alw-cluster
controllerManager: {}
dns:
type: CoreDNS
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
etcd:
external:
endpoints:
- https://etcd-1:2379
- https://etcd-2:2379
- https://etcd-3:2379
caFile: /etc/etcd/pki/ca.crt
certFile: /etc/etcd/pki/apiserver-etcd-client.crt
keyFile: /etc/etcd/pki/apiserver-etcd-client.key
EOF
下载镜像
kubeadm config images pull --config kubeadm-config.yaml
下载镜像脚本
#!/bin/bash
images=(
kube-apiserver:v1.20.4
kube-controller-manager:v1.20.4
kube-scheduler:v1.20.4
kube-proxy:v1.20.4
pause:3.2
etcd:3.4.13-0
coredns:1.7.0)
for image in ${images[@]}; do
# docker
#docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
#docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
#docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
# cri
ctr -n k8s.io i tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
done
初始化
kubeadm init --config kubeadm-config.yaml --upload-certs
初始化成功会出现以下信息
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
--control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256
然后集群二和集群三 运行命令加入
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
--control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d
查看集群
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 NotReady control-plane,master 11m v1.20.5
k8s-master2 NotReady control-plane,master 8m54s v1.20.5
k8s-master3 NotReady control-plane,master 8m31s v1.20.5
由于还未部署网络插件（cni）,状态还是 NotReady
-
注意：使用阿里云 SLB 时，由于apiserver 还未启动，所以此时 6443 端口并未监听，使用 SLB地址时会无法安装成功。修改成当前服务器地址安装成功后再切换 HOST 地址为 SLB地址
本机地址 172.18.20.44 负载均衡地址： 172.18.20.55 172.18.20.44 k8s-master-slb 注意：token有效期为24小时，失效后请在主节点使用以下命令重新生成
kubeadm token create --print-join-command
5.4、部署工作节点
安装 containerd ,kubeadm,kubelet,kubectl 等
工作节点运行 join
kubeadm join k8s-master-slb:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:12b4fe0053bafd6b1b0e05482912b44ddcf88d1d1429e3c611d109ad5bf93ac0
master 查看节点
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 6m28s v1.20.5
k8s-master2 NotReady control-plane,master 3m25s v1.20.5
k8s-master3 NotReady control-plane,master 3m11s v1.20.5
k8s-worker1 NotReady <none> 2m28s v1.20.5
k8s-worker2 NotReady <none> 4s v1.20.5
由于没有部署 CNI 网络插件，状态还是 NotReady
5.5、部署 CNI 网络插件
master 节点运行
需确保kube-flannel.yml文件里的 "Network": "10.10.0.0/16"IP内容与 kube-controller-manager.conf 配置的 --cluster-cidr 一致
即 kubeadm-config 里的 podSubnet
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# kubectl get pods -n kube-system
再次查看节点
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 18h v1.20.5
k8s-master2 Ready control-plane,master 18h v1.20.5
k8s-master3 Ready control-plane,master 18h v1.20.5
k8s-worker1 Ready <none> 18h v1.20.5
k8s-worker2 Ready <none> 18h v1.20.5
5.6、Kubelet驱逐策略优化
修改工作节点kubelet启动参数，更改Pod驱逐策略
vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="EVICTION_HARD=--eviction-hard=memory.available<2Gi,nodefs.available<5Gi,imagefs.available<100Gi"
Environment="EVICTION_RECLAIM=--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=1Gi,imagefs.available=2Gi"
重启kubelet容器，并查看kubelet进程启动参数
systemctl daemon-reload && systemctl restart kubelet
6、单独部署coredns
不依赖kubeadm的方式，适用于不是使用kubeadm创建的k8s集群，或者kubeadm初始化集群之后，删除了dns相关部署。
# 在calico网络中也配置一个coredns # 10.96.0.10 为k8s官方指定的kube-dns地址
mkdir coredns && cd coredns
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
chmod +x deploy.sh
./deploy.sh -i 10.96.0.10 > coredns.yml
kubectl apply -f coredns.yml
# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system
7、Kubernetes 云管理控制器
自建 k8s 想要使用阿里云的负载均衡、存储等服务时，需要部署阿里云提供的组件 cloud-controller-manager
github：https://github.com/kubernetes/cloud-provider-alibaba-cloud
帮助文档：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md
7.1、安装Alibaba CloudProvider 组件
修改 kubelet 启动服务
修改kubelet 启动参数，添加 --cloud-provider=external，并且在kubelet中添加
--hostname-override=${REGION_ID}.${INSTANCE_ID} --provider-id=${REGION_ID}.${INSTANCE_ID}
获取 region-id 和 instance-id
echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`
# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=external --hostname-override=cn-shenzhen.xxxxxxx --provider-id=cn-shenzhen.xxxxxxx
配置阿里云 AccessKeyID，AccessKeySecret
AccessKey & AccessKeySecret 必须以 base64 方式
# base64 AccessKey & AccessKeySecret
$ echo -n "$AccessKeyID" |base64
$ echo -n "$AcceessKeySecret"|base64
$ cat <<EOF >cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-config
namespace: kube-system
data:
cloud-config.conf: |-
{
"Global": {
"accessKeyID": "$your-AccessKeyID-base64",
"accessKeySecret": "$your-AccessKeySecret-base64"
}
}
EOF
$ kubectl create -f cloud-config.yaml
添加 kubeconfig 配置文件(所有master节点)
vim /etc/kubernetes/cloud-controller-manager.conf
kind: Config
contexts:
- context:
cluster: alw-cluster
user: system:cloud-controller-manager
name: system:cloud-controller-manager@alw-cluster
current-context: system:cloud-controller-manager@alw-cluster
users:
- name: system:cloud-controller-manager
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: $CA_DATA
server: https://172.18.20.44:6443
name: alw-cluster
$CA_DATA 可以使用命令 cat /etc/kubernetes/pki/ca.crt|base64 -w 0 获取。
server： 修改为本机IP
编写 cloud-controller-manager.yaml 文件
也可以使用官方推荐的更详细配置 cloud-controller-manager.yml
# vim cloud-controller-manager.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:cloud-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: cloud-controller-manager
tier: control-plane
name: cloud-controller-manager
namespace: kube-system
spec:
selector:
matchLabels:
app: cloud-controller-manager
tier: control-plane
template:
metadata:
labels:
app: cloud-controller-manager
tier: control-plane
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: cloud-controller-manager
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
nodeSelector:
node-role.kubernetes.io/master: ""
containers:
- command:
- /cloud-controller-manager
- --kubeconfig=/etc/kubernetes/cloud-controller-manager.conf
- --address=127.0.0.1
- --allow-untagged-cloud=true
- --leader-elect=true
- --cloud-provider=alicloud # Add your own cloud provider here!
- --use-service-account-credentials=true
- --cloud-config=/etc/kubernetes/config/cloud-config.conf
- --configure-cloud-routes=true
- --allocate-node-cidrs=true
- --route-reconciliation-period=3m
# replace ${cluster-cidr} with your own cluster cidr
- --cluster-cidr=10.10.0.0/16
image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.339-g9830b58-aliyun
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10258
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 15
name: cloud-controller-manager
resources:
requests:
cpu: 200m
volumeMounts:
- mountPath: /etc/kubernetes/
name: k8s
- mountPath: /etc/ssl/certs
name: certs
- mountPath: /etc/pki
name: pki
- mountPath: /etc/kubernetes/config
name: cloud-config
hostNetwork: true
volumes:
- hostPath:
path: /etc/kubernetes
name: k8s
- hostPath:
path: /etc/ssl/certs
name: certs
- hostPath:
path: /etc/pki
name: pki
- configMap:
defaultMode: 420
items:
- key: cloud-config.conf
path: cloud-config.conf
name: cloud-config
name: cloud-config
配置解析
- --cloud-provider=alicloud 云服务商
- --cluster-cidr 集群 pod 地址
运行
# kubectl apply -f cloud-controller-manager.yaml
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cloud-controller-manager-7jbzc 1/1 Running 0 109m
cloud-controller-manager-dfpkv 1/1 Running 0 109m
cloud-controller-manager-lqvtz 1/1 Running 4 109m
完成组件的部署后，接下来就可以使用阿里云的负载均衡了
8、部署 Ingress-nginx
安装 ingress-nginx 控制器
下载：
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml
国内无法访问 google 的镜像仓库，需自己推一个到阿里云镜像仓库并设置为公有
# docker pull k8s.gcr.io/ingress-nginx/controller:v0.45.0
# docker tag k8s.gcr.io/ingress-nginx/controller:v0.45.0 registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
# docker login --username=**** --password=**** registry.cn-shenzhen.aliyuncs.com
# docker push registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
修改配置
...
apiVersion: apps/v1
kind: Deployment
……
# 国内无法访问 google 的镜像仓库，需自己推一个到阿里云镜像仓库并设置为公有
image: registry-vpc.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
……
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-3.27.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.45.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
annotations:
# 指明SLB实例地址类型为私网类型
# service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
# 修改为您的私网SLB实例ID
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: *********-cn-shenzhen-st3-a01
# 是否自动创建SLB端口监听（会覆写已有端口监听），也可手动创建端口监听
#service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: 'true'
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
运行
# kubectl apply -f deploy.yaml
9、部署Aliyun存储插件 csi
9.1、官方ack csi插件
阿里云官方文档：https://help.aliyun.com/document_detail/134722.html?spm=a2c4g.11186623.6.822.7c525ccfsHWlPe
github：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver
RBAC 安装：
下载RBAC配置文件到操作机，并部署：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/rbac.yaml
执行：
$ kubectl apply -f rbac.yaml
CSI-Plugin 安装：
下载普通模版
1. 下载模板：
下载最新版本的CSI Plugin部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-plugin.yaml
将部署模板下载到您的操作机，并保存（csi-plugin.yaml）。
2. 适配模板并部署：
根据集群所在的Region修改模板中的镜像地址。例如：如果是cn-beijing的集群
则将 registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0 中的：
registry 改为 registry-vpc
cn-hangzhou 改为 cn-shenzhen
即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0
模板中的其他镜像也是如此更新；
执行部署：
$ kubectl apply -f csi-plugin.yaml
3. 检查安装情况：
$ kubectl get pod -nkube-system | grep csi-plugin
$ kubectl describe ds csi-plugin -nkube-system | grep Image
CSI-Provisioner 安装：
1. 下载模板：
下载最新版本的CSI Provisioner部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-provisioner.yaml
将部署模板下载到您的操作机，并保存（csi-provisioner.yaml）。
2. 适配模板并部署：
根据集群所在的Region修改模板中的镜像地址。例如：如果是cn-beijing的集群
则将 registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun 中的：
registry 改为 registry-vpc
cn-hangzhou 改为 cn-shenzhen
即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun
模板中的其他镜像也是如此更新；
执行部署：
$ kubectl apply -f csi-provisioner.yaml
3. 检查安装情况：
$ kubectl get pod -nkube-system | grep csi-provisioner
$ kubectl describe deploy csi-provisioner -nkube-system | grep Image
10、部署Dashboard
下载部署文件：
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.0/aio/deploy/recommended.yaml
默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：
# vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001
type: NodePort
selector:
k8s-app: kubernetes-dashboard
部署：
# kubectl apply -f recommended.yaml
# kubectl get pods,svc -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-6b4884c9d5-mjl66 1/1 Running 0 23h
pod/kubernetes-dashboard-7bfbb48676-frmsf 1/1 Running 0 23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.0.0.173 <none> 8000/TCP 23h
service/kubernetes-dashboard NodePort 10.0.0.145 <none> 443:30001/TCP 23h
访问地址：https://NodeIP:30001
创建service account并绑定默认cluster-admin管理员集群角色：
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
查看 token：
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
使用输出的 token 登录 Dashboard
结语
至此，一个生成环境可用的，基于阿里云ecs的k8s集群就搭建完成。但集群的搭建只是基础，后续的维护使用才是重点，包括prometheus监控，istio等
