安装k8s Master高可用集群

主机角色组件

172.18.6.101 K8S Master Kubelet,kubectl,cni,etcd

172.18.6.102 K8S Master Kubelet,kubectl,cni,etcd

172.18.6.103 K8S Master Kubelet,kubectl,cni,etcd

172.18.6.104 K8S Worker Kubelet,cni

172.18.6.105 K8S Worker Kubelet,cni

172.18.6.106 K8S Worker Kubelet,cni

etcd安装

保证k8smaster高可用，不建议使用container的方式启动etcd集群，因为container可能会出现随时死掉的情况，etcd每个节点的启动service又是有状态的。因此此处将以二进制方式进行部署，建议在正式环境中最少部署3个节点的etcd集群，etcd具体安装步骤参考本地服务方式搭建etcd集群

必要组件以及证书安装

ca证书

参考kubernetes中证书生成创建CA证书，并将ca-key.pem与ca.pem放置到k8s集群中所有节点下的/etc/kubernetes/ssl下

woker证书制作

参考kubernetes中证书生成从节点证书生成段落，进行worker节点证书生成。对应ip的证书放置到对应worker节点的/etc/kubernetes/ssl下

kubelet.conf配置安装

创建/etc/kubernetes/kubelet.conf内容如下：

apiVersion: v1

kind: Config

clusters:

- name: local

cluster:

server: https://[负载均衡IP]:[apiserver端口]

certificate-authority: /etc/kubernetes/ssl/ca.pem

users:

- name: kubelet

user:

client-certificate: /etc/kubernetes/ssl/worker.pem

client-key: /etc/kubernetes/ssl/worker-key.pem

contexts:

- context:

cluster: local

user: kubelet

current-context: kubelet-context

cni插件安装

从containernetworking的cni项目中下载cni的必须二进制文件，需要放置到k8s集群中所有节点下的/opt/cni/bin下。

后续将提供rpm包进行一键安装。

kubelet服务部署

注意：后续将提供rpm包进行一键安装。

将对应版本的kubelet二进制文件放置到k8s集群中所有节点下的/usr/bin下

创建/etc/systemd/system/kubelet.service内容如下：

# /etc/systemd/system/kubelet.service

[Unit]

Description=kubelet: The Kubernetes Node Agent

Documentation=http://kubernetes.io/docs/

[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"

Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

Environment="KUBELET_DNS_ARGS=--cluster-dns=10.100.0.10 --cluster-domain=cluster.local"

Environment="KUBELET_EXTRA_ARGS=--pod-infra-container-image=registry.aliyuncs.com/shenshouer/pause-amd64:3.0"

ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS

Restart=always

StartLimitInterval=0

RestartSec=10

[Install]

WantedBy=multi-user.target

创建如下目录：

/etc/kubernetes/

|-- kubelet.conf

|-- manifests

`-- ssl

|-- ca-key.pem

|-- ca.pem

|-- worker.csr

|-- worker-key.pem

|-- worker-openssl.cnf

`-- worker.pem

master组件安装

配置负载均衡

配置LVS使用VIP172.18.6.254指向后端172.18.6.101、172.18.6.102、172.18.6.103，如需简单，则可使用nginx进行TCP4层的负载。

证书生成

openssl.cnf内容如下：

[req]

req_extensions = v3_req

distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = @alt_names

[alt_names]

DNS.1 = kubernetes

DNS.2 = kubernetes.default

DNS.3 = kubernetes.default.svc

DNS.4 = kubernetes.default.svc.cluster.local

DNS.5 = test.example.com.cn

IP.1 = 10.96.0.1

IP.2 = 172.18.6.101

IP.3 = 172.18.6.102

IP.3 = 172.18.6.103

IP.4 = 172.18.6.254

# 三个master的IP

IP.2 = 172.18.6.101

IP.3 = 172.18.6.102

IP.3 = 172.18.6.103

# LVS负载均衡的VIP

IP.4 = 172.18.6.254

# 可能会用到的负载均衡domain

DNS.5 = test.example.com.cn

证书生成具体步骤请参考kubernetes中证书生成 Master证书生成部分与Worker证书生成部分，生成后的证书需要放置到三台Master节点对应路径上

其他组件安装

Master节点上/etc/kubernetes/manifests下放置如下三个文件

kube-apiserver.manifest:

# /etc/kubernetes/manifests/kube-apiserver.manifest

{

"kind": "Pod",

"apiVersion": "v1",

"metadata": {

"name": "kube-apiserver",

"namespace": "kube-system",

"creationTimestamp": null,

"labels": {

"component": "kube-apiserver",

"tier": "control-plane"

}

"spec": {

"volumes": [

{

"name": "k8s",

"hostPath": {

"path": "/etc/kubernetes"

}

{

"name": "certs",

"hostPath": {

"path": "/etc/ssl/certs"

}

"containers": [

{

"name": "kube-apiserver",

"image": "registry.aliyuncs.com.cn/shenshouer/kube-apiserver:v1.5.2",

"command": [

"kube-apiserver",

"--insecure-bind-address=127.0.0.1",

"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",

"--service-cluster-ip-range=10.96.0.0/12",

"--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

"--client-ca-file=/etc/kubernetes/ssl/ca.pem",

"--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",

"--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

"--secure-port=6443",

"--allow-privileged",

"--advertise-address=[当前Master节点IP]",

"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",

"--anonymous-auth=false",

"--etcd-servers=http://127.0.0.1:2379"

"resources": {

"requests": {

"cpu": "250m"

}

"volumeMounts": [

{

"name": "k8s",

"readOnly": true,

"mountPath": "/etc/kubernetes/"

{

"name": "certs",

"mountPath": "/etc/ssl/certs"

}

"livenessProbe": {

"httpGet": {

"path": "/healthz",

"port": 8080,

"host": "127.0.0.1"

"initialDelaySeconds": 15,

"timeoutSeconds": 15,

"failureThreshold": 8

}

"hostNetwork": true

"status": {}

}kube-controller-manager.manifest

{

"kind": "Pod",

"apiVersion": "v1",

"metadata": {

"name": "kube-controller-manager",

"namespace": "kube-system",

"creationTimestamp": null,

"labels": {

"component": "kube-controller-manager",

"tier": "control-plane"

}

"spec": {

"volumes": [

{

"name": "k8s",

"hostPath": {

"path": "/etc/kubernetes"

}

{

"name": "certs",

"hostPath": {

"path": "/etc/ssl/certs"

}

"containers": [

{

"name": "kube-controller-manager",

"image": "registry.aliyuncs.com/shenshouer/kube-controller-manager:v1.5.2",

"command": [

"kube-controller-manager",

"--address=127.0.0.1",

"--leader-elect",

"--master=127.0.0.1:8080",

"--cluster-name=kubernetes",

"--root-ca-file=/etc/kubernetes/ssl/ca.pem",

"--service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

"--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem",

"--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem",

"--insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap",

"--allocate-node-cidrs=true",

"--cluster-cidr=10.244.0.0/16"

"resources": {

"requests": {

"cpu": "200m"

}

"volumeMounts": [

{

"name": "k8s",

"readOnly": true,

"mountPath": "/etc/kubernetes/"

{

"name": "certs",

"mountPath": "/etc/ssl/certs"

}

"livenessProbe": {

"httpGet": {

"path": "/healthz",

"port": 10252,

"host": "127.0.0.1"

"initialDelaySeconds": 15,

"timeoutSeconds": 15,

"failureThreshold": 8

}

"hostNetwork": true

"status": {}

}

kube-scheduler.manifest

{

"kind": "Pod",

"apiVersion": "v1",

"metadata": {

"name": "kube-scheduler",

"namespace": "kube-system",

"creationTimestamp": null,

"labels": {

"component": "kube-scheduler",

"tier": "control-plane"

}

"spec": {

"containers": [

{

"name": "kube-scheduler",

"image": "registry.aliyuncs.com/shenshouer/kube-scheduler:v1.5.2",

"command": [

"kube-scheduler",

"--address=127.0.0.1",

"--leader-elect",

"--master=127.0.0.1:8080"

"resources": {

"requests": {

"cpu": "100m"

}

"livenessProbe": {

"httpGet": {

"path": "/healthz",

"port": 10251,

"host": "127.0.0.1"

"initialDelaySeconds": 15,

"timeoutSeconds": 15,

"failureThreshold": 8

}

"hostNetwork": true

"status": {}

其他组件安装

kube-proxy安装

在任意master上执行kubectl create -f kube-proxy-ds.yaml，其中kube-proxy-ds.yaml内容如下：

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

labels:

component: kube-proxy

k8s-app: kube-proxy

kubernetes.io/cluster-service: "true"

tier: node

namespace: kube-system

spec:

selector:

matchLabels:

component: kube-proxy

k8s-app: kube-proxy

kubernetes.io/cluster-service: "true"

tier: node

template:

metadata:

labels:

component: kube-proxy

k8s-app: kube-proxy

kubernetes.io/cluster-service: "true"

tier: node

spec:

containers:

- command:

- kube-proxy

- --kubeconfig=/run/kubeconfig

- --cluster-cidr=10.244.0.0/16

image: registry.aliyuncs.com/shenshouer/kube-proxy:v1.5.2

imagePullPolicy: IfNotPresent

resources: {}

securityContext:

privileged: true

terminationMessagePath: /dev/termination-log

volumeMounts:

- mountPath: /var/run/dbus

- mountPath: /run/kubeconfig

- mountPath: /etc/kubernetes/ssl

dnsPolicy: ClusterFirst

hostNetwork: true

restartPolicy: Always

securityContext: {}

terminationGracePeriodSeconds: 30

volumes:

- hostPath:

path: /etc/kubernetes/kubelet.conf

- hostPath:

path: /var/run/dbus

- hostPath:

path: /etc/kubernetes/ssl

网络组件安装

在任意master上执行kubectl apply -f kube-flannel.yaml，其中kube-flannel.yaml内容如下，注意，如果是在vagrant启动的虚拟机中运行，请修改flannled启动参数将--iface指向具体通讯网卡

---

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: kube-system

---

kind: ConfigMap

apiVersion: v1

metadata:

namespace: kube-system

labels:

tier: node

app: flannel

data:

cni-conf.json: |

{

"name": "cbr0",

"type": "flannel",

"delegate": {

"ipMasq": true,

"bridge": "cbr0",

"hairpinMode": true,

"forceAddress": true,

"isDefaultGateway": true

}

net-conf.json: |

{

"Network": "10.244.0.0/16",

"Backend": {

"Type": "vxlan"

}

---

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

namespace: kube-system

labels:

tier: node

app: flannel

spec:

template:

metadata:

labels:

tier: node

app: flannel

spec:

hostNetwork: true

nodeSelector:

beta.kubernetes.io/arch: amd64

serviceAccountName: flannel

containers:

- name: kube-flannel

image: registry.aliyuncs.com/shenshouer/flannel:v0.7.0

command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth0" ]

securityContext:

privileged: true

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

volumeMounts:

- name: run

mountPath: /run

- name: flannel-cfg

mountPath: /etc/kube-flannel/

- name: install-cni

image: registry.aliyuncs.com/shenshouer/flannel:v0.7.0

command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]

volumeMounts:

- name: cni

mountPath: /etc/cni/net.d

- name: flannel-cfg

mountPath: /etc/kube-flannel/

volumes:

- name: run

hostPath:

path: /run

- name: cni

hostPath:

path: /etc/cni/net.d

- name: flannel-cfg

configMap:

DNS部署

在任意master上执行kubectl create -f skydns.yaml，其中skydns.yaml内容如下

apiVersion: v1

kind: Service

metadata:

namespace: kube-system

labels:

k8s-app: kube-dns

kubernetes.io/cluster-service: "true"

kubernetes.io/name: "KubeDNS"

spec:

selector:

k8s-app: kube-dns

clusterIP: 10.100.0.10

ports:

- name: dns

port: 53

protocol: UDP

- name: dns-tcp

port: 53

protocol: TCP

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: kube-system

labels:

k8s-app: kube-dns

kubernetes.io/cluster-service: "true"

spec:

# replicas: not specified here:

# 1. In order to make Addon Manager do not reconcile this replicas parameter.

# 2. Default is 1.

# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.

strategy:

rollingUpdate:

maxSurge: 10%

maxUnavailable: 0

selector:

matchLabels:

k8s-app: kube-dns

template:

metadata:

labels:

k8s-app: kube-dns

annotations:

scheduler.alpha.kubernetes.io/critical-pod: ''

scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'

spec:

containers:

- name: kubedns

image: registry.aliyuncs.com/shenshouer/kubedns-amd64:1.9

resources:

# TODO: Set memory limits when we've profiled the container for large

# clusters, then set request = limit to keep this container in

# guaranteed class. Currently, this container falls into the

# "burstable" category so the kubelet doesn't backoff from restarting it.

limits:

memory: 170Mi

requests:

cpu: 100m

memory: 70Mi

livenessProbe:

httpGet:

path: /healthz-kubedns

port: 8080

scheme: HTTP

initialDelaySeconds: 60

timeoutSeconds: 5

successThreshold: 1

failureThreshold: 5

readinessProbe:

httpGet:

path: /readiness

port: 8081

scheme: HTTP

# we poll on pod startup for the Kubernetes master service and

# only setup the /readiness HTTP server once that's available.

initialDelaySeconds: 3

timeoutSeconds: 5

args:

- --domain=cluster.local.

- --dns-port=10053

- --config-map=kube-dns

# This should be set to v=2 only after the new image (cut from 1.5) has

# been released, otherwise we will flood the logs.

- --v=0

- --federations=myfederation=federation.test

env:

- name: PROMETHEUS_PORT

value: "10055"

ports:

- containerPort: 10053

protocol: UDP

- containerPort: 10053

protocol: TCP

- containerPort: 10055

protocol: TCP

- name: dnsmasq

image: registry.aliyuncs.com/shenshouer/kube-dnsmasq-amd64:1.4

livenessProbe:

httpGet:

path: /healthz-dnsmasq

port: 8080

scheme: HTTP

initialDelaySeconds: 60

timeoutSeconds: 5

successThreshold: 1

failureThreshold: 5

args:

- --cache-size=1000

- --no-resolv

- --server=127.0.0.1#10053

- --log-facility=-

ports:

- containerPort: 53

protocol: UDP

- containerPort: 53

protocol: TCP

# see: https://github.com/kubernetes/kubernetes/issues/29055 for details

resources:

requests:

cpu: 150m

memory: 10Mi

- name: dnsmasq-metrics

image: registry.aliyuncs.com/shenshouer/dnsmasq-metrics-amd64:1.0

livenessProbe:

httpGet:

path: /metrics

port: 10054

scheme: HTTP

initialDelaySeconds: 60

timeoutSeconds: 5

successThreshold: 1

failureThreshold: 5

args:

- --v=2

- --logtostderr

ports:

- containerPort: 10054

protocol: TCP

resources:

requests:

memory: 10Mi

- name: healthz

image: registry.aliyuncs.com/shenshouer/exechealthz-amd64:1.2

resources:

limits:

memory: 50Mi

requests:

cpu: 10m

# Note that this container shouldn't really need 50Mi of memory. The

# limits are set higher than expected pending investigation on #29688.

# The extra memory was stolen from the kubedns container to keep the

# net memory requested by the pod constant.

memory: 50Mi

args:

- --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null

- --url=/healthz-dnsmasq

- --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null

- --url=/healthz-kubedns

- --port=8080

- --quiet

ports:

- containerPort: 8080

protocol: TCP

dnsPolicy: Default # Don't use cluster DNS.

Node节点安装

Docker安装

新建/etc/kubernetes/目录

|-- kubelet.conf

|-- manifests

`-- ssl

|-- ca-key.pem

|-- ca.pem

|-- ca.srl

|-- worker.csr

|-- worker-key.pem

|-- worker-openssl.cnf

`-- worker.pem

新建/etc/kubernetes/kubelet.conf配置，参考kubelet.conf配置

新建/etc/kubernetes/ssl，证书制作参考worker证书制作

新建/etc/kubernetes/manifests

新建/opt/cni/bin,安装CNI参考cni安装步骤

安装kubelet,参考kubelet安装

systemctl enable kubelet && systemctl restart kubelet && journalctl -fu kubelet

最后编辑于：2017.12.10 11:00:36

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 206,013评论 6赞 481
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 88,205评论 2赞 382
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 152,370评论 0赞 342
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 55,168评论 1赞 278
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 64,153评论 5赞 371
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 48,954评论 1赞 283
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 38,271评论 3赞 399
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 36,916评论 0赞 259
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 43,382评论 1赞 300
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 35,877评论 2赞 323
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 37,989评论 1赞 333
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,624评论 4赞 322
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 39,209评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 30,199评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,418评论 1赞 260
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 45,401评论 2赞 352
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,700评论 2赞 345

安装k8s Master高可用集群

推荐阅读更多精彩内容