前言

尝到k8s甜头以后，我们就想着应用到生产环境里去，以提高业务迭代效率，可是部署在生产环境里有一个要求，就是k8s集群不可以存在单点故障。。。诶唷我的乖乖，这不就要求k8s集群高可用吗，好，下面就是介绍两种目前比较火的k8s集群master高可用方式。

介绍

首先介绍的第一种k8sHA集群我觉得更应该叫做主从结构k8s集群，它由三台master组成，有三个keepalived提供一个vip 来作为apiserver的ip入口，keepalived设置权重，使得vip落在权重大的master节点上，node节点通过访问这个vip从而访问到这一台master，另外两台master则通过etcd集群，来完成数据同步。

缺点：这样的集群是通过keepalived来实现高可用的，也就是说在权重较大的节点没有故障之前，keepalived所指向的流量永远都是经过主master，只有当主master出现故障或者宕机的情况下，才有可能转移到另外两台从master节点上。这样会导致主master节点压力过大，而另外两台从master可能永远不会被调用，导致资源浪费等等情况。

不过，这也是排除单点故障的一种方式。

下面是理想的高可用架构图。

k8s 理想HA高可用

本文中要部署高可用的架构图：

本文高可用架构

上图摘抄至https://www.kubernetes.org.cn/3536.html

好了，到此我们整理一下本文中需要使用的技术栈

keepalived+etcd+k8s master

其中keepalived提供vip供node做apiserver入口，etcd必须是高可用集群，实现数据同步；以及基本的k8s master节点部署。

安装准备

节点部署相关情况

软件版本：

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

以上软件在上一篇初阶k8s集群搭建里已经介绍并附有下载地址。

环境配置

systemctl stop firewalldsystemctl disable firewalld

修改每个节点hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

keepalived安装

libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

wget http://www.keepalived.org/software/keepalived-1.4.3tar.gz

yum install -y libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

yum -y install libnl libnl-devel

tar -xzvf keepalived-1.4.3.tar.gz

cd keepalived-1.4.3

./configure --prefix=/usr/local/keepalived #检查环境配置

出现上图即为正确环境，如果出现错误

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking foropenssl/ssl.h... no

configure: error: 

  !!! OpenSSL is not properly installed on your system. !!!

  !!! Can not include OpenSSL headers files.            !!!

则：安装openssl和openssl-devel包，然后从新编译配置文件。

yum install openssl openssl-devel

./configure --prefix=/usr/local/keepalived

make && make install

cp keepalived/etc/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ 

ps -aux |grep keepalived

chkconfig keepalived on

通过systemctl status keepalived查看keepalived状态

三台master重复以上步骤，直到完成keepalived的安装。

安装完成后编写配置文件：

master1的keepalived.conf

cat >/etc/keepalived/keepalived.conf <<EOF

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state MASTER

    interface ens33 #本机物理网卡名字,可通过ip a来查看

    virtual_router_id 61

    priority 120  # 主节点权重最高 依次减少

    advert_int 1

    mcast_src_ip 192.168.100.1  #修改为本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #注释掉本地IP 

        #192.168.100.1

        192.168.100.2

        192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

    }

}

EOF

master2的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state BACKUP

    interface ens33 #本机物理网卡名字,可通过ip a来查看

    virtual_router_id 61

    priority 110  # 主节点权重最高 依次减少

    advert_int 1

    mcast_src_ip 192.168.100.2  #修改为本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #注释掉本地IP 

        192.168.100.1

        #192.168.100.2

        192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

    }

}

EOF

master3的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state BACKUP

    interface ens33 #本机物理网卡名字,可通过ip a来查看

    virtual_router_id 61

    priority 100  # 主节点权重最高 依次减少

    advert_int 1

    mcast_src_ip 192.168.100.3  #修改为本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #注释掉本地IP 

        192.168.100.1

        192.168.100.2

        #192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

    }

}

EOF

启动keepalived

systemctl restart keepalived

通过ip a可以查看

除了本机ip还多了一个虚拟ip

也可以通过ping ip去验证vip是否生效。

安装ETCD

1：设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoexport PATH=/usr/local/bin:$PATH

2：创建 CA 配置文件（下面配置的IP为etc节点的IP

mkdir /root/ssl

cd /root/ssl

cat >  ca-config.json <<EOF

{"signing": {"default": { "expiry": "8760h"},"profiles": { "kubernetes-Soulmate": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" }}}}EOF

cat >  ca-csr.json <<EOF

{"CN": "kubernetes-Soulmate","key": {"algo": "rsa","size": 2048},"names": [{ "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System"}]}EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF

{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.100.1", "192.168.100.2", "192.168.100.3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System" } ]}EOF

cfssl gencert -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

3:master1分发etcd证书到master2、master3上面

mkdir -p /etc/etcd/ssl

cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/

ssh -n master2 "mkdir -p /etc/etcd/ssl && exit"

ssh -n master3 "mkdir -p /etc/etcd/ssl && exit"

scp -r /etc/etcd/ssl/*.pem master2:/etc/etcd/ssl/

scp -r /etc/etcd/ssl/*.pem master3:/etc/etcd/ssl/

解压etcd-v3.3.2-linux-amd64.tar.gz并安装

wget https://github.com/coreos/etcd/releases/download/v3.3.2/etcd-v3.3.2-linux-amd64.tar.gz

tar -xzvf etcd-v3.3.2-linux-amd64.tar.gz

cd etcd-v3.3.2-linux-amd64

cp etcd* /bin/

#查看是否安装好

etcd -version 

etcd Version: 3.3.2Git SHA: c9d46ab37Go Version: go1.9.4Go OS/Arch: linux/amd64

etcdctl -version

etcdctl version: 3.3.2API version: 2

在每一个master上都创建一个etcd存储目录mkdir -p /u03/etcd/

这里可以自行选择储存数据地址，但是要记得在下面配置文件中做修改

master1

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

  --name master1 \

  --cert-file=/etc/etcd/ssl/etcd.pem \

  --key-file=/etc/etcd/ssl/etcd-key.pem \

  --peer-cert-file=/etc/etcd/ssl/etcd.pem \

  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \

  --trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --initial-advertise-peer-urls https://192.168.100.1:2380 \

  --listen-peer-urls https://192.168.100.1:2380 \

  --listen-client-urls https://192.168.100.1:2379,http://127.0.0.1:2379 \

  --advertise-client-urls https://192.168.100.1:2379 \

  --initial-cluster-token etcd-cluster-0 \

  --initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

  --initial-cluster-state new \

  --data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

master2

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

  --name master2 \

  --cert-file=/etc/etcd/ssl/etcd.pem \

  --key-file=/etc/etcd/ssl/etcd-key.pem \

  --peer-cert-file=/etc/etcd/ssl/etcd.pem \

  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \

  --trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --initial-advertise-peer-urls https://192.168.100.2:2380 \

  --listen-peer-urls https://192.168.100.2:2380 \

  --listen-client-urls https://192.168.100.2:2379,http://127.0.0.1:2379 \

  --advertise-client-urls https://192.168.220.146:2379 \

  --initial-cluster-token etcd-cluster-0 \

  --initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

  --initial-cluster-state new \

  --data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

master3

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

  --name master3 \

  --cert-file=/etc/etcd/ssl/etcd.pem \

  --key-file=/etc/etcd/ssl/etcd-key.pem \

  --peer-cert-file=/etc/etcd/ssl/etcd.pem \

  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \

  --trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

  --initial-advertise-peer-urls https://192.168.100.3:2380 \

  --listen-peer-urls https://192.168.100.3:2380 \

  --listen-client-urls https://192.168.100.3:2379,http://127.0.0.1:2379 \

  --advertise-client-urls https://192.168.100.3:2379 \

  --initial-cluster-token etcd-cluster-0 \

  --initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

  --initial-cluster-state new \

  --data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

每个master都执行以下命令以启动etcd集群

cd /etc/systemd/system/

mv etcd.service /usr/lib/systemd/system/

systemctl daemon-reload

systemctl enable etcd

systemctl restart etcd

systemctl status etcd

通过以下命令检测集群是否正常

etcdctl --endpoints=https://192.168.100.1:2379,https://192.168.100.2:2379,https://192.168.100.3:2379 \

  --ca-file=/etc/etcd/ssl/ca.pem \

  --cert-file=/etc/etcd/ssl/etcd.pem \

  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

keepalived+etcd安装完成后，开始部署k8s

安装docker、k8s相关rpm包，以及上传k8s相关镜像。请看我上一篇初阶k8s集群搭建。

所有节点修改kubelet配置文件

sed -i -e 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

重启kubelet

systemctl daemon-reload && systemctl restart kubelet

初始化集群，创建集群配置文件

# 生成token

# 保留token后面还要使用

我们使用coreDNS作为k8s集群内部DNS解析，使用canal作为网络服务

kubeadm token generate

cat <<EOF > config.yaml

apiVersion: kubeadm.k8s.io/v1alpha1

kind: MasterConfiguration

etcd:

  endpoints:

  - https://192.168.100.1:2379

  - https://192.168.100.2:2379

  - https://192.168.100.3:2379

  caFile: /etc/etcd/ssl/ca.pem

  certFile: /etc/etcd/ssl/etcd.pem

  keyFile: /etc/etcd/ssl/etcd-key.pem

  dataDir: /var/lib/etcd

networking:

  podSubnet: 10.244.0.0/16

kubernetesVersion: 1.10.0

api:

  advertiseAddress: "192.168.150.186"

token: "hpobow.vw1g1ya5dre7sq06" #刚刚保存的token

tokenTTL: "0s"#表示永不过期

apiServerCertSANs:

- master1

- master2

- master3

- 192.168.100.1

- 192.168.100.2

- 192.168.100.3

- 192.168.100.4

featureGates:

  CoreDNS: true

EOF

编辑完成后执行kubeadm init --config config.yaml

如果失败则查看错误journalctl -xeu kubelet 查看服务启动日志或根据相关日志查看问题

通过kubeadm reset重置

注意，如果etcd已经写入数据，请先到etcd存储数据路径下清空数据记录。

若成功，你会看到以下内容

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node

as root:

kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315

按照上面提示，此时root用户还不能使用kubelet控制集群需要，配置下环境变量

对于非root用户

mkdir -p $HOME/.kube

 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

 sudo chown $(id -u):$(id -g) $HOME/.kube/config

对于root用户

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source一下环境变量

source ~/.bash_profile

kubeadm生成证书密码文件分发到master2和master3上面去

scp -r /etc/kubernetes/pki master2:/etc/kubernetes/

scp -r /etc/kubernetes/pki master3:/etc/kubernetes/

部署canal网络，在master1执行

kubectl apply -f \

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml

kubectl apply-f \https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml

可能镜像下载需要一点时间，也可以先将yaml文件下载到本地，自行修改镜像路径，使用自己下载好的镜像

wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml

wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml

等待部署好了之后查看当前节点是否准备好

[root@master1 ~]# kubectl get node

NAME STATUS ROLES AGE VERSION

master1    Ready    master    31m      v1.10.0

通过kubectl get pods --all-namespaces查看是否所有的容器都已经运行，如果出现error或crash，就使用kubectl describe pod -n kube-system来查看出现的问题。

在master2和master3上面分别执行初始化

使用之前在master1执行的配置config.yaml在另外两个节点上执行kubeadm init --config config.yaml，将获得与master1一样的结果

同样的配置下环境变量

对于非root用户

mkdir -p $HOME/.kube

 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

 sudo chown $(id -u):$(id -g) $HOME/.kube/config

对于root用户

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source一下环境变量

source ~/.bash_profile

[root@master1 ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

master1    Ready    master    1h        v1.10.0

master2    Ready    master    1h        v1.10.0

master3    Ready    master    1h        v1.10.0

查看所有节点运行的容器kubectl get pods --all-namespaces -o wide

这样，基本的主备模式的高可用就搭建完成了，若要部署dashboard请看我上一篇文章初阶k8s集群搭建，值得注意的是，设置dashboard的basicauth的方式进行apiserver的验证，这个设置需要在每一台master上执行以保证高可用。

另外，在k8s 1.10中想使用HPA需要在每个master节点 /etc/kubernetes/manifests/kube-controller-manager.yaml中增加 - --horizontal-pod-autoscaler-use-rest-clients=false 才可以监控到cpu使用率来完成自动扩容。

监控插件heapster

需要有heapster.yaml、influxdb.yaml、grafana.yaml

vim heapster.yaml

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: heapster

  namespace: kube-system

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: heapster

subjects:

  - kind: ServiceAccount

    name: heapster

    namespace: kube-system

roleRef:

  kind: ClusterRole

  name: system:heapster

  apiGroup: rbac.authorization.k8s.io

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: heapster

  namespace: kube-system

spec:

  replicas: 1

  template:

    metadata:

      labels:

        task: monitoring

        k8s-app: heapster

    spec:

      serviceAccountName: heapster

      containers:

      - name: heapster

        image: 192.168.220.84/third_party/heapster-amd64:v1.3.0 #这里我用的是自己的私服镜像地址

        imagePullPolicy: IfNotPresent

        command:

        - /heapster

        - --source=kubernetes:https://kubernetes.default

        - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086

---

apiVersion: v1

kind: Service

metadata:

  labels:

    task: monitoring

    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

    # If you are NOT using this as an addon, you should comment out this line.

    kubernetes.io/cluster-service: 'true'

    kubernetes.io/name: Heapster

  name: heapster

  namespace: kube-system

spec:

  ports:

  - port: 80

    targetPort: 8082

  selector:

    k8s-app: heapster

vim influxdb.yaml

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: monitoring-influxdb

  namespace: kube-system

spec:

  replicas: 1

  template:

    metadata:

      labels:

        task: monitoring

        k8s-app: influxdb

    spec:

      containers:

      - name: influxdb

        image: 192.168.220.84/third_party/heapster-influxdb-amd64:v1.1.1 #私服地址，需要自行更换

        volumeMounts:

        - mountPath: /data

          name: influxdb-storage

      volumes:

      - name: influxdb-storage

        emptyDir: {}

---

apiVersion: v1

kind: Service

metadata:

  labels:

    task: monitoring

    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

    # If you are NOT using this as an addon, you should comment out this line.

    kubernetes.io/cluster-service: 'true'

    kubernetes.io/name: monitoring-influxdb

  name: monitoring-influxdb

  namespace: kube-system

spec:

  ports:

  - port: 8086

    targetPort: 8086

  selector:

    k8s-app: influxdb

vim grafana.yaml

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: monitoring-grafana

  namespace: kube-system

spec:

  replicas: 1

  template:

    metadata:

      labels:

        task: monitoring

        k8s-app: grafana

    spec:

      containers:

      - name: grafana

        image: 192.168.220.84/third_party/heapster-grafana-amd64:v4.4.1 #私服地址，需自行更换

        ports:

        - containerPort: 3000

          protocol: TCP

        volumeMounts:

        - mountPath: /var

          name: grafana-storage

        env:

        - name: INFLUXDB_HOST

          value: monitoring-influxdb

        - name: GF_SERVER_HTTP_PORT

          value: "3000"

          # The following env variables are required to make Grafana accessible via

          # the kubernetes api-server proxy. On production clusters, we recommend

          # removing these env variables, setup auth for grafana, and expose the grafana

          # service using a LoadBalancer or a public IP.

        - name: GF_AUTH_BASIC_ENABLED

          value: "false"

        - name: GF_AUTH_ANONYMOUS_ENABLED

          value: "true"

        - name: GF_AUTH_ANONYMOUS_ORG_ROLE

          value: Admin

        - name: GF_SERVER_ROOT_URL

          # If you're only using the API Server proxy, set this value instead:

          # value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/

          value: /

      volumes:

      - name: grafana-storage

        emptyDir: {}

---

apiVersion: v1

kind: Service

metadata:

  labels:

    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

    # If you are NOT using this as an addon, you should comment out this line.

    kubernetes.io/cluster-service: 'true'

    kubernetes.io/name: monitoring-grafana

  name: monitoring-grafana

  namespace: kube-system

spec:

  # In a production setup, we recommend accessing Grafana through an external Loadbalancer

  # or through a public IP.

  # type: LoadBalancer

  # You could also use NodePort to expose the service at a randomly-generated port

  type: NodePort

  ports:

  - port: 80

    targetPort: 3000

    nodePort: 31236

  selector:

    k8s-app: grafana

执行kubectl apply -f heapster.yaml -f influxdb.yaml -f grafana.yaml

在dashboard上的展示效果

heapster展示

heapster展示

grafana展示

加入的node工作节点

安装以下软件版本，文章开头有说道

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

环境配置

systemctl stop firewalldsystemctl disable firewalld

修改每个节点hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

然后执行在master上留下的kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315即可。

主要参考文章：

kubeadm安装Kubernetes V1.10集群详细文档

kubernetes1.9离线部署