k8s环境搭建
文档介绍在Docker for mac中的k8s如何把环境搭建好。
清单：
- docker for mac中启动k8s
- k8s网络环境介绍
- ingress-nginx安装（网关）
- k8s-dashboard安装（官方提供的UI管理）
- weavescope安装（集群监控）
- mysql安装（事例）
通过以上清单，可以完成大部分日常开发工作。
docker for mac中启动k8s
要启动docker for mac中的k8s，需要下载很多k8s相关镜像，这部分镜像国内网络无法下载，我会列出来所有的镜像可以自己从其他途径下载，也可以直接通过我给的脚本下载。
镜像下载：
#!/bin/bash
kube_version=v1.15.5
kube_proxy_version=$kube_version
kube_controller_manager_version=$kube_version
kube_scheduler_version=$kube_version
kube_apiserver_version=$kube_version
coredns_version=1.3.1
pause_version=3.1
etcd_version=3.3.10
# 镜像列表的前面的是需要下载的镜像，等号后面的是能够正常下载下来的镜像
image_list=(
"k8s.gcr.io/kube-proxy:${kube_proxy_version}=gotok8s/kube-proxy:${kube_proxy_version}"
"k8s.gcr.io/kube-controller-manager:${kube_controller_manager_version}=gotok8s/kube-controller-manager:${kube_controller_manager_version}"
"k8s.gcr.io/kube-scheduler:${kube_scheduler_version}=gotok8s/kube-scheduler:${kube_scheduler_version}"
"k8s.gcr.io/kube-apiserver:${kube_apiserver_version}=gotok8s/kube-apiserver:${kube_apiserver_version}"
"k8s.gcr.io/coredns:${coredns_version}=gotok8s/coredns:${coredns_version}"
"k8s.gcr.io/pause:${pause_version}=gotok8s/pause:${pause_version}"
"k8s.gcr.io/etcd:${etcd_version}=gotok8s/etcd:${etcd_version}"
)
OLD_IFS="$IFS"
for value in "${image_list[@]}"; do
IFS="="
read -r -a image <<<"$value"
image_name=${image[0]}
repeat_image_name=${image[1]}
echo "pull $image_name from $repeat_image_name"
docker pull "$repeat_image_name"
docker tag "$repeat_image_name" "$image_name"
docker rmi "$repeat_image_name"
done
IFS="$OLD_IFS"
下载好了后直接通过docker for mac打开就好了， 大概等5分钟左右：
image.png
k8s单节点集群就安装完成了。
k8s网络环境介绍
k8s中的网络不通比较难找问题，下面先澄清一下：
- docker for mac中宿主机是无法直接访问容器的。具体详见：https://docs.docker.com/docker-for-mac/networking/
- 容器是可以直接访问宿主机的，容器里面有一个docker0的桥接网卡。
ingress-nginx安装
有了k8s集群，集群里面部署的应用我们没办法直接访问，除非使用nodePort的服务形式暴露出来，这样暴露出来有个问题就是：我们直接通过pod所在的机器ip和端口来访问，每次重启应用可能会在不同的机器上，这样就导致访问的ip一直处于变化中。为了解决这个问题，我们私有自建集群就可以通过以下几种方式来解决：
MetaLB。和提供给云厂商的负载均衡一样的。
-
ingress。Ingress 公开了从集群外部到集群内service的HTTP和HTTPS路由。 流量路由由 Ingress 资源上定义的规则控制。
internet | [ Ingress ] --|-----|-- [ Services ]
以上两种是比较推荐来解决ip漂移问题的方案。
安装ingress
- 安装ingress-controller。下载文件：https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml保存成ingress-nginx.yaml
kubectl apply -f ingress-nginx.yaml
这里可以直接把这个文件下载到本机，里面有一个镜像：quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0这个镜像在国内同样可能无法下载，需要自己去下载下来。对于这个版本，提供一个替代下载地址：registry.cn-shenzhen.aliyuncs.com/yangqiang-pub/kubernetes-ingress-controller:0.28.0可以把文件里面的镜像替换成这个地址。
- 安装service。
kubectl apply -f service.yaml
我们主要通过NodePort的方式暴露出去，还有其他选择参考：https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
service.yaml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: NodePort
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 30000
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 30001
这里的端口号必须是30000-32767。所以我们这里直接把30000和30001当成80以及443来使用。尽管使用--service-node-port-rangeAPI服务器标志重新配置NodePort范围以包含非特权端口并能够公开端口80和443。这样做可能会导致意外问题。
到此ingress-nginx就安装完成了。
k8s-dashboard安装（官方提供的UI管理）
-
安装dashboard。如果镜像下载不下来，也要替换。
kubectl apply -f dashboard.yamldashboard.yaml
# Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # 修改说明：修改两个镜像的拉起策略都为imagePullPolicy: IfNotPresent 防止每次从远程拉不到镜像的问题 apiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard --- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard --- kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf: "" --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque --- kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.0.0-rc5 imagePullPolicy: IfNotPresent ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kubernetes-dashboard # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "beta.kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' spec: containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.3 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "beta.kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {} -
安装好后还无法访问，需要创建账号和绑定角色。
kubectl apply -f admin-role.yamladmin-role.yaml
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: admin annotations: rbac.authorization.kubernetes.io/autoupdate: "true" roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: admin namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: admin namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile -
启动。执行以下命令，然后访问：
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/kubectl proxy这样可以直接访问，稍后会使用ingress配置来访问。
-
配置通过ingress启动。
kubectl apply -f ingress.yamlingress.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: kubernetes-dashboard namespace: kubernetes-dashboard annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" spec: tls: - hosts: # 访问的域名 - dashboard.yangqiang.im # 证书名称 secretName: yangqiang.im rules: - host: dashboard.yangqiang.im http: paths: - path: / backend: serviceName: kubernetes-dashboard servicePort: 443创建证书查看教程：https://yangqiang.im/?p=745
-
在k8s集群中创建证书。
kubectl create secret tls yangqiang.im --cert=/Users/carlton/Documents/Ssl/yangqiang.im/fullchain.cer --key=/Users/carlton/Documents/Ssl/yangqiang.im/yangqiang.im.key -n kubernetes-dashboard 因为你的域名没办法直接指向到你自己的机器。所以这里在hosts中新增：
127.0.0.1 dashboard.yangqiang.im。然后通过地址：https://dashboard.yangqiang.im就能直接访问了。
注意:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"这个配置是对应到pod提供的服务是否是https还是http的。当前应用就是提供的443所以用HTTS，一般我们自己的应用都是80或者其他端口这里需要修改成nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
weavescope安装（集群监控）
weavescope是一个k8s集群监控软件，贴个图：
image.png
这个安装很简单，就不贴细节了。官网安装参考：https://www.weave.works/docs/scope/latest/installing/#k8s
ingress配置：
kubectl apply -f ingress.yaml
ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: weave-scope
namespace: weave
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
rules:
- host: weave-scope.yangqiang.im
http:
paths:
- path: /
backend:
serviceName: weave-scope-app
servicePort: app
可以通过：http://weave-scope.yangqiang.im来访问，同样hosts新增：127.0.0.1 weave-scope.yangqiang.im
mysql安装
因为mysql是走的tcp端口，ingress-nginx配置tcp不是标准协议，所以拿来示范如何配置tcp。
- 安装mysql。
kubectl apply -f mysql.yaml
mysql.yaml
kind: Deployment
apiVersion: apps/v1beta2
metadata:
labels:
app: mysql
name: mysql
spec:
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: pvc
hostPath:
# 这里是mysql数据存储的地方，是宿主机上的地址
path: /Users/carlton/Documents/K8s/data
containers:
- name: mysql
image: mysql:5.7.13
volumeMounts:
- name: pvc
mountPath: "/var/lib/mysql"
subPath: "mysql/data"
- name: pvc
mountPath: "/etc/mysql/conf.d"
subPath: "mysql/conf.d"
ports:
- containerPort: 3306
protocol: TCP
env:
- name: TZ
value: 'Asia/Shanghai'
- name: MYSQL_ROOT_PASSWORD
value: "123456"
-
安装service。
kubectl apply -f service.yamlservice.yaml
kind: Service apiVersion: v1 metadata: name: mysql labels: app: mysql spec: type: ClusterIP ports: - name: mysql-3306 port: 3306 targetPort: 3306 protocol: TCP selector: app: mysql -
配置ingress。tcp或者udp服务不需要安装ingress，因为不是标准的协议。如果要通过ingress访问，则需要在ingress-controller应用里面配置。
- 配置服务转发。找到ingress部署文件
ingress-nginx.yaml，然后更新：
kubectl apply -f ingress-nginx.yamlingress-nginx.yaml
...... kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: 30100: "default/mysql:3306" ......注意：
30100: "default/mysql:3306"这个的意思是，default命名空间下面的mysql应用提供的服务端口是3306。然后ingress通过30100端口往外暴露。-
配置ingress的服务暴露给外网，修改ingress的
service.yaml文件kubectl apply -f service.yamlservice.yaml
kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: externalTrafficPolicy: Local type: NodePort selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ports: - name: http port: 80 protocol: TCP targetPort: http nodePort: 30000 - name: https port: 443 protocol: TCP targetPort: https nodePort: 30001 # 在ingress-controller的config-map中配置了30100 -> 3306。这里是暴露30100 # 整个过程：mysql容器c暴露3306，然后mysql服务s也是暴露的3306，ingress配置"30100": "dev/mysql:3306（服务暴露的端口）"，ingerss服务拿着30100暴露30100给宿主。 # c3306->s3306->ingress_30100->外网宿主机30100 - name: mysql-3306 # 随便一个只要不冲突就可以，这个是ingress-controller的服务向集群暴露端口 port: 30100 protocol: TCP # 这个必须是config-map中配置的对应端口 targetPort: 30100 # 这个是主机暴露给外网的端口 nodePort: 30100
到此为止，我们可以通过127.0.0.1:30100来直接连接到mysql数据库。这是pod的tcp端口暴露的方式。
- 配置服务转发。找到ingress部署文件
